2024-05-04T17:29:04.979 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-04T17:29:04.979 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} --archive /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130 -- /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/orig.config.yaml 2024-05-04T17:29:05.018 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-04T17:29:05.099 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130 branch: wip-yuriw11-testing-20240501.200505-squid description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} email: yweinste@redhat.com first_in_suite: false job_id: '7687130' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuriw11-testing-20240501.200505-squid ceph: conf: client: debug cephsqlite: 20 debug client: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 1500 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - POOL_APP_NOT_ENABLED - do not have an application enabled mon_bind_msgr2: false sha1: f273489c6dcd4bc88409993babd09dd99491162c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f273489c6dcd4bc88409993babd09dd99491162c selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuriw11-testing-20240501.200505-squid sha1: f273489c6dcd4bc88409993babd09dd99491162c owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8797 sha1: f273489c6dcd4bc88409993babd09dd99491162c sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuriw11-testing-20240501.200505-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f273489c6dcd4bc88409993babd09dd99491162c targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCnwtQ3UjX+ovj6EqFBJ7I+26RDiS5MVlG5G0jhIn8se7gSyirglsz7eaEGZBORk40HF4wjMixPVeqB7c5I2CY= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8t/NqQ6azIROAkQMLsXeHZC4bcS+tYulmkYG2am70Vu8aF97HazWAv0e+WT8ymGt3Xz3hNSBcMUs36kwJFufQ= tasks: - install: extra_system_packages: deb: - sqlite3 rpm: - sqlite-devel - ceph: null - exec: client.0: - ceph osd pool create cephsqlite - ceph auth get-or-create client.libcephsqlite mon 'profile simple-rados-client-with-blocklist' osd 'allow rwx pool=cephsqlite' >> /etc/ceph/ceph.keyring - exec: client.0: - ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr - workunit: clients: client.0: - rados/test_libcephsqlite.sh cephsqlite env: CEPH_ARGS: --id libcephsqlite --no-log-to-stderr teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-02_23:59:28 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-04T17:29:05.099 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa; will attempt to use it 2024-05-04T17:29:05.100 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks 2024-05-04T17:29:05.100 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-04T17:29:05.102 INFO:teuthology.task.internal:Checking packages... 2024-05-04T17:29:05.126 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f273489c6dcd4bc88409993babd09dd99491162c' 2024-05-04T17:29:05.126 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-04T17:29:05.126 INFO:teuthology.packaging:ref: None 2024-05-04T17:29:05.127 INFO:teuthology.packaging:tag: None 2024-05-04T17:29:05.127 INFO:teuthology.packaging:branch: wip-yuriw11-testing-20240501.200505-squid 2024-05-04T17:29:05.127 INFO:teuthology.packaging:sha1: f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:29:05.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuriw11-testing-20240501.200505-squid 2024-05-04T17:29:05.356 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2497.gf273489c 2024-05-04T17:29:05.357 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-04T17:29:05.367 INFO:teuthology.task.internal:no buildpackages task found 2024-05-04T17:29:05.367 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-04T17:29:05.444 INFO:teuthology.task.internal:Saving configuration 2024-05-04T17:29:05.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-04T17:29:05.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T17:29:05.491 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 17:24:13.341588', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCnwtQ3UjX+ovj6EqFBJ7I+26RDiS5MVlG5G0jhIn8se7gSyirglsz7eaEGZBORk40HF4wjMixPVeqB7c5I2CY='} 2024-05-04T17:29:05.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 17:24:13.342959', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8t/NqQ6azIROAkQMLsXeHZC4bcS+tYulmkYG2am70Vu8aF97HazWAv0e+WT8ymGt3Xz3hNSBcMUs36kwJFufQ='} 2024-05-04T17:29:05.510 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-04T17:29:05.522 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-05-04T17:29:05.522 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-05-04T17:29:05.522 INFO:teuthology.run_tasks:Running task console_log... 2024-05-04T17:29:05.604 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd14285cf70>, signals=[15]) 2024-05-04T17:29:05.604 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-04T17:29:05.614 INFO:teuthology.task.internal:Opening connections... 2024-05-04T17:29:05.614 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:29:05.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:29:05.698 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:29:05.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:29:05.780 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-04T17:29:06.027 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-05-04T17:29:06.045 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-05-04T17:29:06.045 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-05-04T17:29:06.102 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-04T17:29:06.103 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-04T17:29:06.103 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-04T17:29:06.103 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-05-04T17:29:06.142 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-05-04T17:29:06.160 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-05-04T17:29:06.160 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-05-04T17:29:06.217 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-05-04T17:29:06.217 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-05-04T17:29:06.217 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-05-04T17:29:06.217 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-05-04T17:29:06.217 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-05-04T17:29:06.217 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-04T17:29:06.218 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-04T17:29:06.218 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-05-04T17:29:06.240 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-04T17:29:06.257 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-04T17:29:06.299 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-04T17:29:06.299 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-05-04T17:29:06.302 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-05-04T17:29:06.318 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-04T17:29:06.330 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-04T17:29:06.330 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-05-04T17:29:06.358 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-05-04T17:29:06.399 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-04T17:29:06.452 INFO:teuthology.run_tasks:Running task kernel... 2024-05-04T17:29:06.472 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-04T17:29:06.472 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-04T17:29:06.473 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-04T17:29:06.473 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-04T17:29:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:06.489 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-05-04T17:29:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:06.490 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-05-04T17:29:06.548 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-437.el9.x86_64 2024-05-04T17:29:06.548 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-437.el9.x86_64 2024-05-04T17:29:06.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-05-04T17:29:06.549 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-437.el9.x86_64 2024-05-04T17:29:06.549 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-437.el9.x86_64 2024-05-04T17:29:06.549 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-05-04T17:29:08.790 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-05-04T17:29:08.830 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-05-04T17:29:11.606 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-04T17:29:12.333 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-05-04T17:29:16.089 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.2 MB 00:00 2024-05-04T17:29:16.633 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.2 MB 00:00 2024-05-04T17:29:17.841 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-05-04T17:29:18.001 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 16 kB 00:00 2024-05-04T17:29:18.921 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-05-04T17:29:19.659 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-04T17:29:24.019 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-05-04T17:29:24.202 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:01 ago on Sat 04 May 2024 05:29:23 PM UTC. 2024-05-04T17:29:24.725 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-05-04T17:29:25.875 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-04T17:29:25.875 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-04T17:29:25.967 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-05-04T17:29:25.968 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-05-04T17:29:25.968 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-05-04T17:29:26.065 DEBUG:teuthology.orchestra.run.smithi110:> echo no | sudo yum reinstall kernel || true 2024-05-04T17:29:26.565 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 05:29:23 PM UTC. 2024-05-04T17:29:26.631 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-04T17:29:26.631 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-04T17:29:26.687 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout:Reinstalling: 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:29:26.698 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-05-04T17:29:26.699 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-04T17:29:26.699 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:29:26.699 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 255 k 2024-05-04T17:29:26.699 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 0 2024-05-04T17:29:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:Operation aborted. 2024-05-04T17:29:26.723 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-05-04T17:29:26.724 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-05-04T17:29:26.724 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-05-04T17:29:26.757 INFO:teuthology.orchestra.run.smithi110.stdout:Is this ok [y/N]: 2024-05-04T17:29:26.757 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum reinstall -y kernel || true 2024-05-04T17:29:26.843 DEBUG:teuthology.orchestra.run.smithi136:> echo no | sudo yum reinstall kernel || true 2024-05-04T17:29:27.251 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:04 ago on Sat 04 May 2024 05:29:23 PM UTC. 2024-05-04T17:29:27.341 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 05:29:24 PM UTC. 2024-05-04T17:29:27.369 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:Reinstalling: 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 255 k 2024-05-04T17:29:27.370 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 0 2024-05-04T17:29:27.371 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-05-04T17:29:27.468 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-05-04T17:29:27.477 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout:Reinstalling: 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 255 k 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 0 2024-05-04T17:29:27.478 INFO:teuthology.orchestra.run.smithi136.stderr:Operation aborted. 2024-05-04T17:29:27.545 INFO:teuthology.orchestra.run.smithi136.stdout:Is this ok [y/N]: 2024-05-04T17:29:27.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum reinstall -y kernel || true 2024-05-04T17:29:28.043 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:04 ago on Sat 04 May 2024 05:29:24 PM UTC. 2024-05-04T17:29:28.159 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-05-04T17:29:28.160 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-05-04T17:29:28.160 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:28.160 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout:Reinstalling: 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 255 k 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 0 2024-05-04T17:29:28.161 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-05-04T17:29:28.438 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-437.el9.x86_64.rpm 320 kB/s | 255 kB 00:00 2024-05-04T17:29:28.438 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:29:28.438 INFO:teuthology.orchestra.run.smithi110.stdout:Total 239 kB/s | 255 kB 00:01 2024-05-04T17:29:28.459 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-05-04T17:29:28.643 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-05-04T17:29:28.643 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-05-04T17:29:28.648 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-05-04T17:29:28.649 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-05-04T17:29:28.779 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-05-04T17:29:28.847 INFO:teuthology.orchestra.run.smithi110.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:28.847 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:29.501 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-437.el9.x86_64.rpm 240 kB/s | 255 kB 00:01 2024-05-04T17:29:29.502 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:29:29.502 INFO:teuthology.orchestra.run.smithi136.stdout:Total 190 kB/s | 255 kB 00:01 2024-05-04T17:29:29.525 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-05-04T17:29:29.579 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:29.579 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:29.717 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-05-04T17:29:29.718 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-05-04T17:29:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-05-04T17:29:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-05-04T17:29:29.860 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-05-04T17:29:29.936 INFO:teuthology.orchestra.run.smithi136.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:29.936 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:29:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Reinstalled: 2024-05-04T17:29:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:29:30.318 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-05-04T17:29:30.491 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-05-04T17:29:30.538 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:30.538 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-04T17:29:30.538 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-04T17:29:30.538 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-04T17:29:30.538 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-04T17:29:30.580 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:30.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:30.589 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-05-04T17:29:30.604 DEBUG:teuthology.parallel:result is None 2024-05-04T17:29:31.299 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:31.299 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:29:31.300 INFO:teuthology.orchestra.run.smithi136.stdout:Reinstalled: 2024-05-04T17:29:31.300 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:31.300 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:29:31.300 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-05-04T17:29:31.498 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-05-04T17:29:31.545 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:31.545 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-04T17:29:31.545 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-04T17:29:31.545 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-04T17:29:31.546 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-04T17:29:31.574 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-05-04T17:29:31.593 DEBUG:teuthology.parallel:result is None 2024-05-04T17:29:31.594 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-04T17:29:31.603 INFO:teuthology.task.internal:Creating test directory... 2024-05-04T17:29:31.603 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-04T17:29:31.606 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-04T17:29:31.649 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-04T17:29:31.760 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-04T17:29:31.774 INFO:teuthology.task.internal:Creating archive directory... 2024-05-04T17:29:31.774 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-04T17:29:31.777 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-04T17:29:31.805 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-04T17:29:31.815 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-04T17:29:31.816 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-04T17:29:31.842 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-04T17:29:31.872 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:31.875 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:31.883 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:31.884 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:31.885 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-04T17:29:31.897 INFO:teuthology.task.internal:Configuring sudo... 2024-05-04T17:29:31.898 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-04T17:29:31.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-04T17:29:31.952 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-04T17:29:32.035 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-04T17:29:32.036 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-04T17:29:32.038 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-04T17:29:32.053 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:32.139 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:32.191 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:32.271 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:32.313 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:29:32.313 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-04T17:29:32.376 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:32.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:32.463 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:32.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:32.584 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:29:32.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-04T17:29:32.648 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-05-04T17:29:32.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-05-04T17:29:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T17:29:32.730 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T17:29:32.997 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-04T17:29:33.006 INFO:teuthology.task.internal:Starting timer... 2024-05-04T17:29:33.006 INFO:teuthology.run_tasks:Running task pcp... 2024-05-04T17:29:33.200 INFO:teuthology.run_tasks:Running task selinux... 2024-05-04T17:29:33.362 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-04T17:29:33.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-05-04T17:29:33.403 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-05-04T17:29:33.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-05-04T17:29:33.477 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-05-04T17:29:33.479 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-04T17:29:33.479 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-05-04T17:29:33.501 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-05-04T17:29:33.502 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-05-04T17:29:33.518 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-05-04T17:29:33.518 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-05-04T17:29:33.518 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T17:29:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:33.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T17:29:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:33.590 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-04T17:29:33.590 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-05-04T17:29:33.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-05-04T17:29:33.663 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-04T17:29:33.673 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-04T17:29:33.690 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-04T17:29:33.691 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-04T17:33:23.939 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-05-04T17:33:23.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-04T17:33:23.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:33:24.024 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-05-04T17:33:24.101 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-04T17:33:24.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-04T17:33:24.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:33:24.177 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-05-04T17:33:24.253 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-04T17:33:24.254 INFO:teuthology.run_tasks:Running task clock... 2024-05-04T17:33:24.264 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-04T17:33:24.264 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-04T17:33:24.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T17:33:24.267 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-04T17:33:24.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T17:33:24.292 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-04T17:33:24.305 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-04T17:33:24.321 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-04T17:33:24.327 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-05-04T17:33:24.335 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-04T17:33:24.338 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-05-04T17:33:24.351 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-04T17:33:24.357 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-05-04T17:33:24.364 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-04T17:33:24.370 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-05-04T17:33:24.380 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-04T17:33:24.393 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-04T17:33:24.415 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-05-04T17:33:24.418 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T17:33:24.419 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-05-04T17:33:24.419 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.419 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.419 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.419 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.446 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-05-04T17:33:24.449 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T17:33:24.449 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-05-04T17:33:24.449 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.449 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.449 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.449 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:24.450 INFO:teuthology.run_tasks:Running task install... 2024-05-04T17:33:24.458 DEBUG:teuthology.task.install:project ceph 2024-05-04T17:33:24.458 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}} 2024-05-04T17:33:24.458 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'} 2024-05-04T17:33:24.459 INFO:teuthology.task.install:Using flavor: default 2024-05-04T17:33:24.464 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-04T17:33:24.465 INFO:teuthology.task.install:extra packages: [] 2024-05-04T17:33:24.465 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'tag': None, 'wait_for_package': False} 2024-05-04T17:33:24.465 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:33:24.468 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'tag': None, 'wait_for_package': False} 2024-05-04T17:33:24.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:33:24.617 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/ 2024-05-04T17:33:24.617 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2497.gf273489c 2024-05-04T17:33:24.621 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/ 2024-05-04T17:33:24.621 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2497.gf273489c 2024-05-04T17:33:24.732 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-04T17:33:24.732 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:33:24.732 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-04T17:33:24.757 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, sqlite-devel, sqlite-devel, sqlite-devel on remote rpm x86_64 2024-05-04T17:33:24.758 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f273489c6dcd4bc88409993babd09dd99491162c/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-04T17:33:24.796 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-04T17:33:24.796 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:33:24.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-04T17:33:24.822 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, sqlite-devel, sqlite-devel, sqlite-devel, sqlite-devel on remote rpm x86_64 2024-05-04T17:33:24.822 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f273489c6dcd4bc88409993babd09dd99491162c/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-04T17:33:24.822 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-04T17:33:24.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-04T17:33:24.894 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-04T17:33:24.957 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-05-04T17:33:24.958 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-05-04T17:33:24.959 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-04T17:33:25.022 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-05-04T17:33:25.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-05-04T17:33:25.186 INFO:teuthology.orchestra.run.smithi110.stdout:50 files removed 2024-05-04T17:33:25.212 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd sqlite-devel sqlite-devel sqlite-devel sqlite-devel 2024-05-04T17:33:25.249 INFO:teuthology.orchestra.run.smithi136.stdout:50 files removed 2024-05-04T17:33:25.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd sqlite-devel sqlite-devel sqlite-devel sqlite-devel 2024-05-04T17:33:25.818 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 241 kB/s | 83 kB 00:00 2024-05-04T17:33:25.948 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-05-04T17:33:26.236 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-05-04T17:33:26.324 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-05-04T17:33:26.586 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-04T17:33:26.741 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-05-04T17:33:27.418 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-05-04T17:33:28.636 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:02 2024-05-04T17:33:29.384 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-05-04T17:33:31.302 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-05-04T17:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.2 MB 00:01 2024-05-04T17:33:36.293 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-05-04T17:33:36.696 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-04T17:33:37.974 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-04T17:33:38.997 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-05-04T17:33:40.366 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-04T17:33:43.120 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-04T17:33:45.222 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-04T17:33:45.222 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-04T17:33:45.282 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-2497.gf273489c.el9 ceph 6.6 k 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.4 M 2024-05-04T17:33:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 889 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2497.gf273489c.el9 ceph 145 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.5 M 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 142 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 3.5 M 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 7.4 M 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 49 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 10 M 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-2497.gf273489c.el9 ceph 48 M 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 264 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 763 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 32 k 2024-05-04T17:33:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 745 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 127 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-2497.gf273489c.el9 ceph 162 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-2497.gf273489c.el9 ceph 321 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 298 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 99 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 85 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 170 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: sqlite-devel x86_64 3.34.1-7.el9 appstream 134 k 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.4 M 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:33:45.292 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 21 M 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 32 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-2497.gf273489c.el9 ceph 2.4 M 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 249 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-2497.gf273489c.el9 ceph 4.7 M 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 17 M 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 15 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-2497.gf273489c.el9 ceph 25 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-04T17:33:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-2497.gf273489c.el9 ceph 163 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-04T17:33:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 502 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.3 M 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-04T17:33:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 45 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 135 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-04T17:33:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-04T17:33:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-04T17:33:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-04T17:33:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: sqlite x86_64 3.34.1-7.el9 appstream 748 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-04T17:33:45.300 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout:Install 126 Packages 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-05-04T17:33:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:33:45.302 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 201 M 2024-05-04T17:33:45.302 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-05-04T17:33:45.399 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-04T17:33:46.022 INFO:teuthology.orchestra.run.smithi136.stdout:(1/128): ceph-19.0.0-2497.gf273489c.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-05-04T17:33:46.272 INFO:teuthology.orchestra.run.smithi136.stdout:(2/128): ceph-fuse-19.0.0-2497.gf273489c.el9.x8 3.5 MB/s | 889 kB 00:00 2024-05-04T17:33:46.314 INFO:teuthology.orchestra.run.smithi136.stdout:(3/128): ceph-immutable-object-cache-19.0.0-249 3.4 MB/s | 145 kB 00:00 2024-05-04T17:33:46.697 INFO:teuthology.orchestra.run.smithi136.stdout:(4/128): ceph-mds-19.0.0-2497.gf273489c.el9.x86 6.2 MB/s | 2.4 MB 00:00 2024-05-04T17:33:46.890 INFO:teuthology.orchestra.run.smithi136.stdout:(5/128): ceph-mgr-19.0.0-2497.gf273489c.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-05-04T17:33:47.256 INFO:teuthology.orchestra.run.smithi136.stdout:(6/128): ceph-mon-19.0.0-2497.gf273489c.el9.x86 13 MB/s | 4.7 MB 00:00 2024-05-04T17:33:47.500 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-04T17:33:47.501 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-04T17:33:47.560 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-2497.gf273489c.el9 ceph 6.6 k 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.4 M 2024-05-04T17:33:47.568 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 889 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2497.gf273489c.el9 ceph 145 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.5 M 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 142 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 3.5 M 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 7.4 M 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 49 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 10 M 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-2497.gf273489c.el9 ceph 48 M 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 264 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 763 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 32 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 745 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 127 k 2024-05-04T17:33:47.569 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-2497.gf273489c.el9 ceph 162 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-2497.gf273489c.el9 ceph 321 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 298 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 99 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 85 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 170 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: sqlite-devel x86_64 3.34.1-7.el9 appstream 134 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.4 M 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-04T17:33:47.570 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 21 M 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 32 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-2497.gf273489c.el9 ceph 2.4 M 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 249 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-2497.gf273489c.el9 ceph 4.7 M 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 17 M 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 15 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-2497.gf273489c.el9 ceph 25 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-04T17:33:47.571 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-2497.gf273489c.el9 ceph 163 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 502 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.3 M 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-04T17:33:47.572 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 45 k 2024-05-04T17:33:47.573 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 135 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-04T17:33:47.574 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-04T17:33:47.575 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-04T17:33:47.576 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-04T17:33:47.577 INFO:teuthology.orchestra.run.smithi110.stdout: sqlite x86_64 3.34.1-7.el9 appstream 748 k 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout:Install 126 Packages 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-05-04T17:33:47.578 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:33:47.579 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 201 M 2024-05-04T17:33:47.579 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-05-04T17:33:48.190 INFO:teuthology.orchestra.run.smithi136.stdout:(7/128): ceph-osd-19.0.0-2497.gf273489c.el9.x86 18 MB/s | 17 MB 00:00 2024-05-04T17:33:48.526 INFO:teuthology.orchestra.run.smithi110.stdout:(1/128): ceph-19.0.0-2497.gf273489c.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-05-04T17:33:48.798 INFO:teuthology.orchestra.run.smithi136.stdout:(8/128): ceph-radosgw-19.0.0-2497.gf273489c.el9 17 MB/s | 10 MB 00:00 2024-05-04T17:33:48.849 INFO:teuthology.orchestra.run.smithi136.stdout:(9/128): ceph-selinux-19.0.0-2497.gf273489c.el9 505 kB/s | 25 kB 00:00 2024-05-04T17:33:48.909 INFO:teuthology.orchestra.run.smithi110.stdout:(2/128): ceph-fuse-19.0.0-2497.gf273489c.el9.x8 2.3 MB/s | 889 kB 00:00 2024-05-04T17:33:49.009 INFO:teuthology.orchestra.run.smithi110.stdout:(3/128): ceph-immutable-object-cache-19.0.0-249 1.4 MB/s | 145 kB 00:00 2024-05-04T17:33:49.369 INFO:teuthology.orchestra.run.smithi110.stdout:(4/128): ceph-mds-19.0.0-2497.gf273489c.el9.x86 6.6 MB/s | 2.4 MB 00:00 2024-05-04T17:33:49.424 INFO:teuthology.orchestra.run.smithi136.stdout:(10/128): ceph-base-19.0.0-2497.gf273489c.el9.x 1.5 MB/s | 5.4 MB 00:03 2024-05-04T17:33:49.524 INFO:teuthology.orchestra.run.smithi136.stdout:(11/128): libcephfs-devel-19.0.0-2497.gf273489c 318 kB/s | 32 kB 00:00 2024-05-04T17:33:49.560 INFO:teuthology.orchestra.run.smithi110.stdout:(5/128): ceph-mgr-19.0.0-2497.gf273489c.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-05-04T17:33:50.060 INFO:teuthology.orchestra.run.smithi110.stdout:(6/128): ceph-mon-19.0.0-2497.gf273489c.el9.x86 9.4 MB/s | 4.7 MB 00:00 2024-05-04T17:33:50.074 INFO:teuthology.orchestra.run.smithi136.stdout:(12/128): libcephfs2-19.0.0-2497.gf273489c.el9. 1.3 MB/s | 745 kB 00:00 2024-05-04T17:33:50.283 INFO:teuthology.orchestra.run.smithi136.stdout:(13/128): libcephsqlite-19.0.0-2497.gf273489c.e 785 kB/s | 163 kB 00:00 2024-05-04T17:33:50.492 INFO:teuthology.orchestra.run.smithi136.stdout:(14/128): librados-devel-19.0.0-2497.gf273489c. 608 kB/s | 127 kB 00:00 2024-05-04T17:33:50.942 INFO:teuthology.orchestra.run.smithi136.stdout:(15/128): libradosstriper1-19.0.0-2497.gf273489 1.1 MB/s | 502 kB 00:00 2024-05-04T17:33:51.360 INFO:teuthology.orchestra.run.smithi110.stdout:(7/128): ceph-osd-19.0.0-2497.gf273489c.el9.x86 13 MB/s | 17 MB 00:01 2024-05-04T17:33:52.177 INFO:teuthology.orchestra.run.smithi110.stdout:(8/128): ceph-radosgw-19.0.0-2497.gf273489c.el9 13 MB/s | 10 MB 00:00 2024-05-04T17:33:52.236 INFO:teuthology.orchestra.run.smithi110.stdout:(9/128): ceph-selinux-19.0.0-2497.gf273489c.el9 433 kB/s | 25 kB 00:00 2024-05-04T17:33:53.653 INFO:teuthology.orchestra.run.smithi110.stdout:(10/128): ceph-base-19.0.0-2497.gf273489c.el9.x 997 kB/s | 5.4 MB 00:05 2024-05-04T17:33:53.745 INFO:teuthology.orchestra.run.smithi110.stdout:(11/128): libcephfs-devel-19.0.0-2497.gf273489c 346 kB/s | 32 kB 00:00 2024-05-04T17:33:54.333 INFO:teuthology.orchestra.run.smithi136.stdout:(16/128): librgw2-19.0.0-2497.gf273489c.el9.x86 1.6 MB/s | 5.3 MB 00:03 2024-05-04T17:33:54.378 INFO:teuthology.orchestra.run.smithi110.stdout:(12/128): libcephfs2-19.0.0-2497.gf273489c.el9. 1.1 MB/s | 745 kB 00:00 2024-05-04T17:33:54.442 INFO:teuthology.orchestra.run.smithi136.stdout:(17/128): python3-ceph-argparse-19.0.0-2497.gf2 417 kB/s | 45 kB 00:00 2024-05-04T17:33:54.600 INFO:teuthology.orchestra.run.smithi136.stdout:(18/128): python3-ceph-common-19.0.0-2497.gf273 854 kB/s | 135 kB 00:00 2024-05-04T17:33:54.637 INFO:teuthology.orchestra.run.smithi110.stdout:(13/128): ceph-common-19.0.0-2497.gf273489c.el9 3.2 MB/s | 21 MB 00:06 2024-05-04T17:33:54.662 INFO:teuthology.orchestra.run.smithi110.stdout:(14/128): libcephsqlite-19.0.0-2497.gf273489c.e 576 kB/s | 163 kB 00:00 2024-05-04T17:33:54.726 INFO:teuthology.orchestra.run.smithi136.stdout:(19/128): python3-cephfs-19.0.0-2497.gf273489c. 1.3 MB/s | 162 kB 00:00 2024-05-04T17:33:54.754 INFO:teuthology.orchestra.run.smithi110.stdout:(15/128): librados-devel-19.0.0-2497.gf273489c. 1.1 MB/s | 127 kB 00:00 2024-05-04T17:33:54.993 INFO:teuthology.orchestra.run.smithi136.stdout:(20/128): python3-rados-19.0.0-2497.gf273489c.e 1.2 MB/s | 321 kB 00:00 2024-05-04T17:33:55.071 INFO:teuthology.orchestra.run.smithi110.stdout:(16/128): libradosstriper1-19.0.0-2497.gf273489 1.2 MB/s | 502 kB 00:00 2024-05-04T17:33:55.171 INFO:teuthology.orchestra.run.smithi110.stdout:(17/128): python3-ceph-argparse-19.0.0-2497.gf2 450 kB/s | 45 kB 00:00 2024-05-04T17:33:55.277 INFO:teuthology.orchestra.run.smithi136.stdout:(21/128): python3-rbd-19.0.0-2497.gf273489c.el9 1.0 MB/s | 298 kB 00:00 2024-05-04T17:33:55.330 INFO:teuthology.orchestra.run.smithi110.stdout:(18/128): python3-ceph-common-19.0.0-2497.gf273 853 kB/s | 135 kB 00:00 2024-05-04T17:33:55.385 INFO:teuthology.orchestra.run.smithi136.stdout:(22/128): python3-rgw-19.0.0-2497.gf273489c.el9 918 kB/s | 99 kB 00:00 2024-05-04T17:33:55.452 INFO:teuthology.orchestra.run.smithi136.stdout:(23/128): rbd-fuse-19.0.0-2497.gf273489c.el9.x8 1.3 MB/s | 85 kB 00:00 2024-05-04T17:33:55.772 INFO:teuthology.orchestra.run.smithi110.stdout:(19/128): ceph-test-19.0.0-2497.gf273489c.el9.x 14 MB/s | 48 MB 00:03 2024-05-04T17:33:55.797 INFO:teuthology.orchestra.run.smithi110.stdout:(20/128): python3-cephfs-19.0.0-2497.gf273489c. 346 kB/s | 162 kB 00:00 2024-05-04T17:33:55.857 INFO:teuthology.orchestra.run.smithi110.stdout:(21/128): python3-rados-19.0.0-2497.gf273489c.e 3.7 MB/s | 321 kB 00:00 2024-05-04T17:33:55.923 INFO:teuthology.orchestra.run.smithi110.stdout:(22/128): python3-rgw-19.0.0-2497.gf273489c.el9 1.5 MB/s | 99 kB 00:00 2024-05-04T17:33:55.949 INFO:teuthology.orchestra.run.smithi110.stdout:(23/128): python3-rbd-19.0.0-2497.gf273489c.el9 1.9 MB/s | 298 kB 00:00 2024-05-04T17:33:55.982 INFO:teuthology.orchestra.run.smithi110.stdout:(24/128): rbd-fuse-19.0.0-2497.gf273489c.el9.x8 1.4 MB/s | 85 kB 00:00 2024-05-04T17:33:56.049 INFO:teuthology.orchestra.run.smithi110.stdout:(25/128): rbd-nbd-19.0.0-2497.gf273489c.el9.x86 2.5 MB/s | 170 kB 00:00 2024-05-04T17:33:56.125 INFO:teuthology.orchestra.run.smithi110.stdout:(26/128): librgw2-19.0.0-2497.gf273489c.el9.x86 3.9 MB/s | 5.3 MB 00:01 2024-05-04T17:33:56.150 INFO:teuthology.orchestra.run.smithi110.stdout:(27/128): ceph-grafana-dashboards-19.0.0-2497.g 317 kB/s | 32 kB 00:00 2024-05-04T17:33:56.175 INFO:teuthology.orchestra.run.smithi110.stdout:(28/128): ceph-mgr-cephadm-19.0.0-2497.gf273489 2.8 MB/s | 142 kB 00:00 2024-05-04T17:33:56.277 INFO:teuthology.orchestra.run.smithi136.stdout:(24/128): ceph-test-19.0.0-2497.gf273489c.el9.x 6.5 MB/s | 48 MB 00:07 2024-05-04T17:33:56.377 INFO:teuthology.orchestra.run.smithi136.stdout:(25/128): rbd-nbd-19.0.0-2497.gf273489c.el9.x86 1.7 MB/s | 170 kB 00:00 2024-05-04T17:33:56.412 INFO:teuthology.orchestra.run.smithi136.stdout:(26/128): ceph-grafana-dashboards-19.0.0-2497.g 951 kB/s | 32 kB 00:00 2024-05-04T17:33:56.417 INFO:teuthology.orchestra.run.smithi110.stdout:(29/128): ceph-mgr-dashboard-19.0.0-2497.gf2734 13 MB/s | 3.5 MB 00:00 2024-05-04T17:33:56.462 INFO:teuthology.orchestra.run.smithi136.stdout:(27/128): ceph-mgr-cephadm-19.0.0-2497.gf273489 2.8 MB/s | 142 kB 00:00 2024-05-04T17:33:56.484 INFO:teuthology.orchestra.run.smithi110.stdout:(30/128): ceph-mgr-modules-core-19.0.0-2497.gf2 3.7 MB/s | 249 kB 00:00 2024-05-04T17:33:56.559 INFO:teuthology.orchestra.run.smithi110.stdout:(31/128): ceph-mgr-rook-19.0.0-2497.gf273489c.e 660 kB/s | 49 kB 00:00 2024-05-04T17:33:56.626 INFO:teuthology.orchestra.run.smithi110.stdout:(32/128): ceph-prometheus-alerts-19.0.0-2497.gf 231 kB/s | 15 kB 00:00 2024-05-04T17:33:56.662 INFO:teuthology.orchestra.run.smithi136.stdout:(28/128): rbd-mirror-19.0.0-2497.gf273489c.el9. 2.6 MB/s | 3.1 MB 00:01 2024-05-04T17:33:56.701 INFO:teuthology.orchestra.run.smithi110.stdout:(33/128): ceph-volume-19.0.0-2497.gf273489c.el9 3.4 MB/s | 264 kB 00:00 2024-05-04T17:33:56.802 INFO:teuthology.orchestra.run.smithi110.stdout:(34/128): cephadm-19.0.0-2497.gf273489c.el9.noa 7.5 MB/s | 763 kB 00:00 2024-05-04T17:33:56.879 INFO:teuthology.orchestra.run.smithi136.stdout:(29/128): ceph-mgr-dashboard-19.0.0-2497.gf2734 8.4 MB/s | 3.5 MB 00:00 2024-05-04T17:33:56.919 INFO:teuthology.orchestra.run.smithi110.stdout:(35/128): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-05-04T17:33:56.929 INFO:teuthology.orchestra.run.smithi136.stdout:(30/128): ceph-mgr-modules-core-19.0.0-2497.gf2 4.9 MB/s | 249 kB 00:00 2024-05-04T17:33:56.969 INFO:teuthology.orchestra.run.smithi110.stdout:(36/128): rbd-mirror-19.0.0-2497.gf273489c.el9. 3.0 MB/s | 3.1 MB 00:01 2024-05-04T17:33:56.979 INFO:teuthology.orchestra.run.smithi136.stdout:(31/128): ceph-mgr-rook-19.0.0-2497.gf273489c.e 987 kB/s | 49 kB 00:00 2024-05-04T17:33:57.021 INFO:teuthology.orchestra.run.smithi136.stdout:(32/128): ceph-prometheus-alerts-19.0.0-2497.gf 369 kB/s | 15 kB 00:00 2024-05-04T17:33:57.053 INFO:teuthology.orchestra.run.smithi110.stdout:(37/128): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-05-04T17:33:57.071 INFO:teuthology.orchestra.run.smithi136.stdout:(33/128): ceph-volume-19.0.0-2497.gf273489c.el9 5.2 MB/s | 264 kB 00:00 2024-05-04T17:33:57.095 INFO:teuthology.orchestra.run.smithi110.stdout:(38/128): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-05-04T17:33:57.120 INFO:teuthology.orchestra.run.smithi110.stdout:(39/128): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-05-04T17:33:57.162 INFO:teuthology.orchestra.run.smithi110.stdout:(40/128): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-05-04T17:33:57.186 INFO:teuthology.orchestra.run.smithi136.stdout:(34/128): cephadm-19.0.0-2497.gf273489c.el9.noa 6.5 MB/s | 763 kB 00:00 2024-05-04T17:33:57.187 INFO:teuthology.orchestra.run.smithi110.stdout:(41/128): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-05-04T17:33:57.221 INFO:teuthology.orchestra.run.smithi110.stdout:(42/128): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-04T17:33:57.254 INFO:teuthology.orchestra.run.smithi110.stdout:(43/128): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-04T17:33:57.280 INFO:teuthology.orchestra.run.smithi110.stdout:(44/128): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-05-04T17:33:57.372 INFO:teuthology.orchestra.run.smithi110.stdout:(45/128): ceph-mgr-diskprediction-local-19.0.0- 6.2 MB/s | 7.4 MB 00:01 2024-05-04T17:33:57.555 INFO:teuthology.orchestra.run.smithi110.stdout:(46/128): flexiblas-3.0.4-8.el9.x86_64.rpm 119 kB/s | 33 kB 00:00 2024-05-04T17:33:57.656 INFO:teuthology.orchestra.run.smithi110.stdout:(47/128): flexiblas-openblas-openmp-3.0.4-8.el9 181 kB/s | 18 kB 00:00 2024-05-04T17:33:57.708 INFO:teuthology.orchestra.run.smithi136.stdout:(35/128): fuse-2.9.9-15.el9.x86_64.rpm 154 kB/s | 80 kB 00:00 2024-05-04T17:33:57.714 INFO:teuthology.orchestra.run.smithi110.stdout:(48/128): boost-program-options-1.75.0-8.el9.x8 232 kB/s | 107 kB 00:00 2024-05-04T17:33:57.815 INFO:teuthology.orchestra.run.smithi110.stdout:(49/128): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2024-05-04T17:33:57.924 INFO:teuthology.orchestra.run.smithi110.stdout:(50/128): libpmemobj-1.12.1-1.el9.x86_64.rpm 599 kB/s | 160 kB 00:00 2024-05-04T17:33:58.124 INFO:teuthology.orchestra.run.smithi110.stdout:(51/128): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-05-04T17:33:58.206 INFO:teuthology.orchestra.run.smithi136.stdout:(36/128): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-05-04T17:33:58.232 INFO:teuthology.orchestra.run.smithi110.stdout:(52/128): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-04T17:33:58.258 INFO:teuthology.orchestra.run.smithi110.stdout:(53/128): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-05-04T17:33:58.314 INFO:teuthology.orchestra.run.smithi136.stdout:(37/128): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-05-04T17:33:58.333 INFO:teuthology.orchestra.run.smithi110.stdout:(54/128): openblas-0.3.21-2.el9.x86_64.rpm 351 kB/s | 35 kB 00:00 2024-05-04T17:33:58.423 INFO:teuthology.orchestra.run.smithi136.stdout:(38/128): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-05-04T17:33:58.567 INFO:teuthology.orchestra.run.smithi110.stdout:(55/128): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-05-04T17:33:58.724 INFO:teuthology.orchestra.run.smithi136.stdout:(39/128): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-05-04T17:33:58.757 INFO:teuthology.orchestra.run.smithi136.stdout:(40/128): python3-cryptography-36.0.1-4.el9.x86 3.7 MB/s | 1.2 MB 00:00 2024-05-04T17:33:58.783 INFO:teuthology.orchestra.run.smithi110.stdout:(56/128): protobuf-3.14.0-13.el9.x86_64.rpm 2.2 MB/s | 1.0 MB 00:00 2024-05-04T17:33:58.859 INFO:teuthology.orchestra.run.smithi110.stdout:(57/128): openblas-openmp-0.3.21-2.el9.x86_64.r 7.9 MB/s | 4.7 MB 00:00 2024-05-04T17:33:58.891 INFO:teuthology.orchestra.run.smithi136.stdout:(41/128): python3-pycparser-2.20-6.el9.noarch.r 1.0 MB/s | 135 kB 00:00 2024-05-04T17:33:58.892 INFO:teuthology.orchestra.run.smithi110.stdout:(58/128): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-05-04T17:33:58.994 INFO:teuthology.orchestra.run.smithi110.stdout:(59/128): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-05-04T17:33:59.000 INFO:teuthology.orchestra.run.smithi136.stdout:(42/128): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-05-04T17:33:59.093 INFO:teuthology.orchestra.run.smithi110.stdout:(60/128): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-05-04T17:33:59.108 INFO:teuthology.orchestra.run.smithi136.stdout:(43/128): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-05-04T17:33:59.150 INFO:teuthology.orchestra.run.smithi136.stdout:(44/128): python3-ply-3.11-14.el9.noarch.rpm 250 kB/s | 106 kB 00:00 2024-05-04T17:33:59.201 INFO:teuthology.orchestra.run.smithi110.stdout:(61/128): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-05-04T17:33:59.227 INFO:teuthology.orchestra.run.smithi110.stdout:(62/128): python3-jinja2-2.11.3-5.el9.noarch.rp 674 kB/s | 248 kB 00:00 2024-05-04T17:33:59.234 INFO:teuthology.orchestra.run.smithi136.stdout:(45/128): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-05-04T17:33:59.259 INFO:teuthology.orchestra.run.smithi136.stdout:(46/128): boost-program-options-1.75.0-8.el9.x8 711 kB/s | 107 kB 00:00 2024-05-04T17:33:59.302 INFO:teuthology.orchestra.run.smithi136.stdout:(47/128): flexiblas-openblas-openmp-3.0.4-8.el9 435 kB/s | 18 kB 00:00 2024-05-04T17:33:59.302 INFO:teuthology.orchestra.run.smithi110.stdout:(63/128): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-05-04T17:33:59.360 INFO:teuthology.orchestra.run.smithi136.stdout:(48/128): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-04T17:33:59.411 INFO:teuthology.orchestra.run.smithi110.stdout:(64/128): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-05-04T17:33:59.419 INFO:teuthology.orchestra.run.smithi136.stdout:(49/128): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-05-04T17:33:59.444 INFO:teuthology.orchestra.run.smithi136.stdout:(50/128): librabbitmq-0.11.0-7.el9.x86_64.rpm 540 kB/s | 45 kB 00:00 2024-05-04T17:33:59.469 INFO:teuthology.orchestra.run.smithi136.stdout:(51/128): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-05-04T17:33:59.503 INFO:teuthology.orchestra.run.smithi136.stdout:(52/128): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-04T17:33:59.519 INFO:teuthology.orchestra.run.smithi110.stdout:(65/128): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-05-04T17:33:59.528 INFO:teuthology.orchestra.run.smithi136.stdout:(53/128): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-05-04T17:33:59.554 INFO:teuthology.orchestra.run.smithi136.stdout:(54/128): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-05-04T17:33:59.628 INFO:teuthology.orchestra.run.smithi110.stdout:(66/128): python3-packaging-20.9-5.el9.noarch.r 713 kB/s | 77 kB 00:00 2024-05-04T17:33:59.637 INFO:teuthology.orchestra.run.smithi136.stdout:(55/128): openblas-openmp-0.3.21-2.el9.x86_64.r 44 MB/s | 4.7 MB 00:00 2024-05-04T17:33:59.671 INFO:teuthology.orchestra.run.smithi136.stdout:(56/128): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-05-04T17:33:59.713 INFO:teuthology.orchestra.run.smithi136.stdout:(57/128): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-05-04T17:33:59.737 INFO:teuthology.orchestra.run.smithi110.stdout:(67/128): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-05-04T17:33:59.788 INFO:teuthology.orchestra.run.smithi136.stdout:(58/128): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-05-04T17:33:59.814 INFO:teuthology.orchestra.run.smithi136.stdout:(59/128): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-05-04T17:33:59.830 INFO:teuthology.orchestra.run.smithi136.stdout:(60/128): python3-jsonpatch-1.21-16.el9.noarch. 612 kB/s | 26 kB 00:00 2024-05-04T17:33:59.845 INFO:teuthology.orchestra.run.smithi110.stdout:(68/128): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-05-04T17:33:59.847 INFO:teuthology.orchestra.run.smithi136.stdout:(61/128): python3-jsonpointer-2.0-4.el9.noarch. 572 kB/s | 19 kB 00:00 2024-05-04T17:33:59.873 INFO:teuthology.orchestra.run.smithi136.stdout:(62/128): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-05-04T17:33:59.898 INFO:teuthology.orchestra.run.smithi136.stdout:(63/128): python3-markupsafe-1.1.1-12.el9.x86_6 689 kB/s | 35 kB 00:00 2024-05-04T17:33:59.940 INFO:teuthology.orchestra.run.smithi136.stdout:(64/128): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.9 MB/s | 250 kB 00:00 2024-05-04T17:33:59.954 INFO:teuthology.orchestra.run.smithi110.stdout:(69/128): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-05-04T17:34:00.007 INFO:teuthology.orchestra.run.smithi136.stdout:(65/128): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-05-04T17:34:00.034 INFO:teuthology.orchestra.run.smithi136.stdout:(66/128): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-05-04T17:34:00.058 INFO:teuthology.orchestra.run.smithi136.stdout:(67/128): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-04T17:34:00.062 INFO:teuthology.orchestra.run.smithi110.stdout:(70/128): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-05-04T17:34:00.083 INFO:teuthology.orchestra.run.smithi136.stdout:(68/128): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-05-04T17:34:00.109 INFO:teuthology.orchestra.run.smithi136.stdout:(69/128): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-04T17:34:00.134 INFO:teuthology.orchestra.run.smithi136.stdout:(70/128): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-04T17:34:00.159 INFO:teuthology.orchestra.run.smithi136.stdout:(71/128): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-04T17:34:00.171 INFO:teuthology.orchestra.run.smithi110.stdout:(71/128): python3-pytz-2021.1-5.el9.noarch.rpm 470 kB/s | 51 kB 00:00 2024-05-04T17:34:00.184 INFO:teuthology.orchestra.run.smithi136.stdout:(72/128): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-04T17:34:00.210 INFO:teuthology.orchestra.run.smithi136.stdout:(73/128): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-04T17:34:00.252 INFO:teuthology.orchestra.run.smithi136.stdout:(74/128): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-05-04T17:34:00.280 INFO:teuthology.orchestra.run.smithi110.stdout:(72/128): python3-requests-oauthlib-1.3.0-12.el 496 kB/s | 54 kB 00:00 2024-05-04T17:34:00.294 INFO:teuthology.orchestra.run.smithi136.stdout:(75/128): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-05-04T17:34:00.336 INFO:teuthology.orchestra.run.smithi136.stdout:(76/128): qatlib-service-24.02.0-1.el9.x86_64.r 860 kB/s | 36 kB 00:00 2024-05-04T17:34:00.378 INFO:teuthology.orchestra.run.smithi136.stdout:(77/128): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-04T17:34:00.545 INFO:teuthology.orchestra.run.smithi136.stdout:(78/128): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-05-04T17:34:00.570 INFO:teuthology.orchestra.run.smithi136.stdout:(79/128): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-05-04T17:34:00.604 INFO:teuthology.orchestra.run.smithi136.stdout:(80/128): sqlite-3.34.1-7.el9.x86_64.rpm 13 MB/s | 748 kB 00:00 2024-05-04T17:34:00.629 INFO:teuthology.orchestra.run.smithi136.stdout:(81/128): sqlite-devel-3.34.1-7.el9.x86_64.rpm 2.2 MB/s | 134 kB 00:00 2024-05-04T17:34:00.654 INFO:teuthology.orchestra.run.smithi136.stdout:(82/128): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-04T17:34:00.771 INFO:teuthology.orchestra.run.smithi136.stdout:(83/128): lua-devel-5.4.4-4.el9.x86_64.rpm 158 kB/s | 22 kB 00:00 2024-05-04T17:34:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:(84/128): ceph-common-19.0.0-2497.gf273489c.el9 1.4 MB/s | 21 MB 00:15 2024-05-04T17:34:01.097 INFO:teuthology.orchestra.run.smithi136.stdout:(85/128): protobuf-compiler-3.14.0-13.el9.x86_6 1.9 MB/s | 863 kB 00:00 2024-05-04T17:34:01.281 INFO:teuthology.orchestra.run.smithi136.stdout:(86/128): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-05-04T17:34:01.314 INFO:teuthology.orchestra.run.smithi136.stdout:(87/128): gperftools-libs-2.9.1-2.el9.x86_64.rp 904 kB/s | 309 kB 00:00 2024-05-04T17:34:01.339 INFO:teuthology.orchestra.run.smithi136.stdout:(88/128): abseil-cpp-20211102.0-3.el9.x86_64.rp 965 kB/s | 548 kB 00:00 2024-05-04T17:34:01.398 INFO:teuthology.orchestra.run.smithi136.stdout:(89/128): libarrow-doc-9.0.0-11.el9.noarch.rpm 310 kB/s | 26 kB 00:00 2024-05-04T17:34:01.440 INFO:teuthology.orchestra.run.smithi136.stdout:(90/128): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-05-04T17:34:01.499 INFO:teuthology.orchestra.run.smithi136.stdout:(91/128): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-05-04T17:34:01.532 INFO:teuthology.orchestra.run.smithi136.stdout:(92/128): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-05-04T17:34:01.633 INFO:teuthology.orchestra.run.smithi136.stdout:(93/128): python3-asyncssh-2.13.2-5.el9.noarch. 5.3 MB/s | 548 kB 00:00 2024-05-04T17:34:01.691 INFO:teuthology.orchestra.run.smithi136.stdout:(94/128): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-05-04T17:34:01.717 INFO:teuthology.orchestra.run.smithi136.stdout:(95/128): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-05-04T17:34:01.784 INFO:teuthology.orchestra.run.smithi136.stdout:(96/128): libarrow-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-05-04T17:34:01.809 INFO:teuthology.orchestra.run.smithi136.stdout:(97/128): python3-cachetools-4.2.4-1.el9.noarch 274 kB/s | 32 kB 00:00 2024-05-04T17:34:01.834 INFO:teuthology.orchestra.run.smithi136.stdout:(98/128): python3-certifi-2021.10.8-2.el9.noarc 128 kB/s | 15 kB 00:00 2024-05-04T17:34:01.893 INFO:teuthology.orchestra.run.smithi136.stdout:(99/128): python3-cherrypy-18.6.1-2.el9.noarch. 4.2 MB/s | 358 kB 00:00 2024-05-04T17:34:01.918 INFO:teuthology.orchestra.run.smithi136.stdout:(100/128): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-05-04T17:34:01.969 INFO:teuthology.orchestra.run.smithi136.stdout:(101/128): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-05-04T17:34:02.069 INFO:teuthology.orchestra.run.smithi136.stdout:(102/128): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-05-04T17:34:02.094 INFO:teuthology.orchestra.run.smithi136.stdout:(103/128): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-05-04T17:34:02.119 INFO:teuthology.orchestra.run.smithi136.stdout:(104/128): python3-grpcio-tools-1.46.7-10.el9.x 719 kB/s | 144 kB 00:00 2024-05-04T17:34:02.145 INFO:teuthology.orchestra.run.smithi136.stdout:(105/128): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-05-04T17:34:02.170 INFO:teuthology.orchestra.run.smithi136.stdout:(106/128): python3-jaraco-collections-3.0.0-8.e 306 kB/s | 23 kB 00:00 2024-05-04T17:34:02.195 INFO:teuthology.orchestra.run.smithi136.stdout:(107/128): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-05-04T17:34:02.229 INFO:teuthology.orchestra.run.smithi136.stdout:(108/128): python3-jaraco-text-3.2.0-6.el9.noar 234 kB/s | 20 kB 00:00 2024-05-04T17:34:02.288 INFO:teuthology.orchestra.run.smithi136.stdout:(109/128): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-05-04T17:34:02.313 INFO:teuthology.orchestra.run.smithi136.stdout:(110/128): python3-more-itertools-8.12.0-2.el9. 944 kB/s | 79 kB 00:00 2024-05-04T17:34:02.346 INFO:teuthology.orchestra.run.smithi136.stdout:(111/128): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-05-04T17:34:02.405 INFO:teuthology.orchestra.run.smithi136.stdout:(112/128): python3-natsort-7.1.1-5.el9.noarch.r 492 kB/s | 58 kB 00:00 2024-05-04T17:34:02.430 INFO:teuthology.orchestra.run.smithi136.stdout:(113/128): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-05-04T17:34:02.456 INFO:teuthology.orchestra.run.smithi136.stdout:(114/128): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-05-04T17:34:02.489 INFO:teuthology.orchestra.run.smithi136.stdout:(115/128): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-05-04T17:34:02.531 INFO:teuthology.orchestra.run.smithi136.stdout:(116/128): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-05-04T17:34:02.573 INFO:teuthology.orchestra.run.smithi136.stdout:(117/128): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-04T17:34:02.598 INFO:teuthology.orchestra.run.smithi136.stdout:(118/128): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-05-04T17:34:02.624 INFO:teuthology.orchestra.run.smithi136.stdout:(119/128): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-05-04T17:34:02.666 INFO:teuthology.orchestra.run.smithi136.stdout:(120/128): python3-typing-extensions-4.4.0-2.el 552 kB/s | 51 kB 00:00 2024-05-04T17:34:02.700 INFO:teuthology.orchestra.run.smithi136.stdout:(121/128): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-05-04T17:34:02.741 INFO:teuthology.orchestra.run.smithi136.stdout:(122/128): python3-websocket-client-1.2.3-2.el9 763 kB/s | 90 kB 00:00 2024-05-04T17:34:02.775 INFO:teuthology.orchestra.run.smithi136.stdout:(123/128): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-05-04T17:34:02.792 INFO:teuthology.orchestra.run.smithi136.stdout:(124/128): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-05-04T17:34:02.984 INFO:teuthology.orchestra.run.smithi136.stdout:(125/128): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-05-04T17:34:03.068 INFO:teuthology.orchestra.run.smithi136.stdout:(126/128): re2-20211101-3.el9.x86_64.rpm 588 kB/s | 192 kB 00:00 2024-05-04T17:34:03.218 INFO:teuthology.orchestra.run.smithi136.stdout:(127/128): librados2-19.0.0-2497.gf273489c.el9. 8.0 MB/s | 3.4 MB 00:00 2024-05-04T17:34:03.926 INFO:teuthology.orchestra.run.smithi136.stdout:(128/128): librbd1-19.0.0-2497.gf273489c.el9.x8 3.3 MB/s | 3.1 MB 00:00 2024-05-04T17:34:03.930 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:03.931 INFO:teuthology.orchestra.run.smithi136.stdout:Total 11 MB/s | 201 MB 00:18 2024-05-04T17:34:04.522 INFO:teuthology.orchestra.run.smithi110.stdout:(73/128): python3-babel-2.9.1-2.el9.noarch.rpm 1.0 MB/s | 6.0 MB 00:05 2024-05-04T17:34:04.639 INFO:teuthology.orchestra.run.smithi110.stdout:(74/128): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-05-04T17:34:04.738 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-05-04T17:34:04.842 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-05-04T17:34:04.842 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-05-04T17:34:04.989 INFO:teuthology.orchestra.run.smithi110.stdout:(75/128): qatlib-24.02.0-1.el9.x86_64.rpm 630 kB/s | 221 kB 00:00 2024-05-04T17:34:05.098 INFO:teuthology.orchestra.run.smithi110.stdout:(76/128): qatlib-service-24.02.0-1.el9.x86_64.r 332 kB/s | 36 kB 00:00 2024-05-04T17:34:05.206 INFO:teuthology.orchestra.run.smithi110.stdout:(77/128): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-05-04T17:34:05.632 INFO:teuthology.orchestra.run.smithi110.stdout:(78/128): socat-1.7.4.1-5.el9.x86_64.rpm 717 kB/s | 305 kB 00:00 2024-05-04T17:34:06.274 INFO:teuthology.orchestra.run.smithi110.stdout:(79/128): python3-numpy-1.20.1-5.el9.x86_64.rpm 741 kB/s | 5.1 MB 00:07 2024-05-04T17:34:06.375 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-05-04T17:34:06.375 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-05-04T17:34:06.532 INFO:teuthology.orchestra.run.smithi110.stdout:(80/128): sqlite-devel-3.34.1-7.el9.x86_64.rpm 519 kB/s | 134 kB 00:00 2024-05-04T17:34:06.616 INFO:teuthology.orchestra.run.smithi110.stdout:(81/128): sqlite-3.34.1-7.el9.x86_64.rpm 761 kB/s | 748 kB 00:00 2024-05-04T17:34:06.725 INFO:teuthology.orchestra.run.smithi110.stdout:(82/128): xmlstarlet-1.6.1-20.el9.x86_64.rpm 331 kB/s | 64 kB 00:00 2024-05-04T17:34:07.186 INFO:teuthology.orchestra.run.smithi110.stdout:(83/128): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-05-04T17:34:07.309 INFO:teuthology.orchestra.run.smithi110.stdout:(84/128): abseil-cpp-20211102.0-3.el9.x86_64.rp 4.3 MB/s | 548 kB 00:00 2024-05-04T17:34:07.367 INFO:teuthology.orchestra.run.smithi110.stdout:(85/128): gperftools-libs-2.9.1-2.el9.x86_64.rp 5.2 MB/s | 309 kB 00:00 2024-05-04T17:34:07.392 INFO:teuthology.orchestra.run.smithi110.stdout:(86/128): grpc-data-1.46.7-10.el9.noarch.rpm 785 kB/s | 19 kB 00:00 2024-05-04T17:34:07.534 INFO:teuthology.orchestra.run.smithi110.stdout:(87/128): libarrow-9.0.0-11.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-05-04T17:34:07.568 INFO:teuthology.orchestra.run.smithi110.stdout:(88/128): libarrow-doc-9.0.0-11.el9.noarch.rpm 781 kB/s | 26 kB 00:00 2024-05-04T17:34:07.602 INFO:teuthology.orchestra.run.smithi110.stdout:(89/128): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-05-04T17:34:07.635 INFO:teuthology.orchestra.run.smithi110.stdout:(90/128): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-05-04T17:34:07.677 INFO:teuthology.orchestra.run.smithi110.stdout:(91/128): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-05-04T17:34:07.703 INFO:teuthology.orchestra.run.smithi110.stdout:(92/128): protobuf-compiler-3.14.0-13.el9.x86_6 883 kB/s | 863 kB 00:00 2024-05-04T17:34:07.736 INFO:teuthology.orchestra.run.smithi110.stdout:(93/128): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-05-04T17:34:07.778 INFO:teuthology.orchestra.run.smithi110.stdout:(94/128): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-05-04T17:34:07.812 INFO:teuthology.orchestra.run.smithi110.stdout:(95/128): python3-cachetools-4.2.4-1.el9.noarch 963 kB/s | 32 kB 00:00 2024-05-04T17:34:07.845 INFO:teuthology.orchestra.run.smithi110.stdout:(96/128): python3-certifi-2021.10.8-2.el9.noarc 452 kB/s | 15 kB 00:00 2024-05-04T17:34:07.896 INFO:teuthology.orchestra.run.smithi110.stdout:(97/128): python3-asyncssh-2.13.2-5.el9.noarch. 2.8 MB/s | 548 kB 00:00 2024-05-04T17:34:07.921 INFO:teuthology.orchestra.run.smithi110.stdout:(98/128): python3-cheroot-8.6.0-4.el9.noarch.rp 2.2 MB/s | 172 kB 00:00 2024-05-04T17:34:07.956 INFO:teuthology.orchestra.run.smithi110.stdout:(99/128): python3-google-auth-2.29.0-1.el9.noar 6.0 MB/s | 206 kB 00:00 2024-05-04T17:34:07.988 INFO:teuthology.orchestra.run.smithi110.stdout:(100/128): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-05-04T17:34:08.047 INFO:teuthology.orchestra.run.smithi110.stdout:(101/128): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-05-04T17:34:08.097 INFO:teuthology.orchestra.run.smithi110.stdout:(102/128): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-05-04T17:34:08.123 INFO:teuthology.orchestra.run.smithi110.stdout:(103/128): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-05-04T17:34:08.340 INFO:teuthology.orchestra.run.smithi110.stdout:(104/128): python3-jaraco-classes-3.2.1-5.el9.n 73 kB/s | 18 kB 00:00 2024-05-04T17:34:08.365 INFO:teuthology.orchestra.run.smithi110.stdout:(105/128): python3-jaraco-collections-3.0.0-8.e 96 kB/s | 23 kB 00:00 2024-05-04T17:34:08.390 INFO:teuthology.orchestra.run.smithi110.stdout:(106/128): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-05-04T17:34:08.416 INFO:teuthology.orchestra.run.smithi110.stdout:(107/128): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-05-04T17:34:08.449 INFO:teuthology.orchestra.run.smithi110.stdout:(108/128): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-05-04T17:34:08.475 INFO:teuthology.orchestra.run.smithi110.stdout:(109/128): python3-logutils-0.3.5-21.el9.noarch 791 kB/s | 46 kB 00:00 2024-05-04T17:34:08.500 INFO:teuthology.orchestra.run.smithi110.stdout:(110/128): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-04T17:34:08.525 INFO:teuthology.orchestra.run.smithi110.stdout:(111/128): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-04T17:34:08.559 INFO:teuthology.orchestra.run.smithi110.stdout:(112/128): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-05-04T17:34:08.584 INFO:teuthology.orchestra.run.smithi110.stdout:(113/128): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-05-04T17:34:08.643 INFO:teuthology.orchestra.run.smithi110.stdout:(114/128): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-05-04T17:34:08.668 INFO:teuthology.orchestra.run.smithi110.stdout:(115/128): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-05-04T17:34:08.693 INFO:teuthology.orchestra.run.smithi110.stdout:(116/128): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-05-04T17:34:08.719 INFO:teuthology.orchestra.run.smithi110.stdout:(117/128): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-04T17:34:08.744 INFO:teuthology.orchestra.run.smithi110.stdout:(118/128): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-05-04T17:34:08.769 INFO:teuthology.orchestra.run.smithi110.stdout:(119/128): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-05-04T17:34:08.794 INFO:teuthology.orchestra.run.smithi110.stdout:(120/128): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-05-04T17:34:08.820 INFO:teuthology.orchestra.run.smithi110.stdout:(121/128): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-05-04T17:34:08.862 INFO:teuthology.orchestra.run.smithi110.stdout:(122/128): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-05-04T17:34:08.879 INFO:teuthology.orchestra.run.smithi110.stdout:(123/128): python3-zc-lockfile-2.0-10.el9.noarc 340 kB/s | 20 kB 00:00 2024-05-04T17:34:08.904 INFO:teuthology.orchestra.run.smithi110.stdout:(124/128): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-05-04T17:34:09.022 INFO:teuthology.orchestra.run.smithi110.stdout:(125/128): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-05-04T17:34:09.880 INFO:teuthology.orchestra.run.smithi110.stdout:(126/128): librados2-19.0.0-2497.gf273489c.el9. 3.5 MB/s | 3.4 MB 00:00 2024-05-04T17:34:11.230 INFO:teuthology.orchestra.run.smithi110.stdout:(127/128): librbd1-19.0.0-2497.gf273489c.el9.x8 1.4 MB/s | 3.1 MB 00:02 2024-05-04T17:34:11.562 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-05-04T17:34:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/130 2024-05-04T17:34:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/130 2024-05-04T17:34:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/130 2024-05-04T17:34:12.038 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/130 2024-05-04T17:34:12.148 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 5/130 2024-05-04T17:34:12.253 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 5/130 2024-05-04T17:34:12.282 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 6/130 2024-05-04T17:34:12.357 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 6/130 2024-05-04T17:34:12.399 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 7/130 2024-05-04T17:34:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/130 2024-05-04T17:34:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/130 2024-05-04T17:34:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 10/130 2024-05-04T17:34:12.598 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 10/130 2024-05-04T17:34:12.627 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86 11/130 2024-05-04T17:34:12.690 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86 11/130 2024-05-04T17:34:12.720 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 12/130 2024-05-04T17:34:12.740 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 12/130 2024-05-04T17:34:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 13/130 2024-05-04T17:34:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 13/130 2024-05-04T17:34:13.086 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 14/130 2024-05-04T17:34:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 15/130 2024-05-04T17:34:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 16/130 2024-05-04T17:34:13.255 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 17/130 2024-05-04T17:34:13.329 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/130 2024-05-04T17:34:13.371 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/130 2024-05-04T17:34:13.414 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 20/130 2024-05-04T17:34:13.447 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 21/130 2024-05-04T17:34:13.474 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 22/130 2024-05-04T17:34:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 23/130 2024-05-04T17:34:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 24/130 2024-05-04T17:34:13.644 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/130 2024-05-04T17:34:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 26/130 2024-05-04T17:34:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 27/130 2024-05-04T17:34:13.790 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-2497.gf273489c.el9. 28/130 2024-05-04T17:34:13.834 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el 29/130 2024-05-04T17:34:13.862 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_6 30/130 2024-05-04T17:34:13.935 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 31/130 2024-05-04T17:34:13.986 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 32/130 2024-05-04T17:34:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 33/130 2024-05-04T17:34:14.205 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 34/130 2024-05-04T17:34:14.264 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 35/130 2024-05-04T17:34:14.807 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 36/130 2024-05-04T17:34:14.894 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 37/130 2024-05-04T17:34:14.921 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 38/130 2024-05-04T17:34:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 39/130 2024-05-04T17:34:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_6 40/130 2024-05-04T17:34:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 41/130 2024-05-04T17:34:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 42/130 2024-05-04T17:34:15.164 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 43/130 2024-05-04T17:34:15.195 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 44/130 2024-05-04T17:34:15.231 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 45/130 2024-05-04T17:34:15.278 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 46/130 2024-05-04T17:34:15.324 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 47/130 2024-05-04T17:34:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 48/130 2024-05-04T17:34:15.504 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 49/130 2024-05-04T17:34:15.542 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 50/130 2024-05-04T17:34:15.584 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 51/130 2024-05-04T17:34:15.680 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 52/130 2024-05-04T17:34:15.717 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 53/130 2024-05-04T17:34:15.749 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 54/130 2024-05-04T17:34:15.774 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 55/130 2024-05-04T17:34:15.799 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 56/130 2024-05-04T17:34:15.875 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 57/130 2024-05-04T17:34:16.003 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 58/130 2024-05-04T17:34:16.043 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 59/130 2024-05-04T17:34:16.079 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 60/130 2024-05-04T17:34:16.121 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : sqlite-3.34.1-7.el9.x86_64 61/130 2024-05-04T17:34:16.156 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 62/130 2024-05-04T17:34:16.186 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 63/130 2024-05-04T17:34:16.220 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 64/130 2024-05-04T17:34:16.254 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 65/130 2024-05-04T17:34:16.295 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 66/130 2024-05-04T17:34:16.322 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 67/130 2024-05-04T17:34:16.402 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 68/130 2024-05-04T17:34:16.452 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 69/130 2024-05-04T17:34:16.483 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 70/130 2024-05-04T17:34:16.531 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 71/130 2024-05-04T17:34:16.630 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 72/130 2024-05-04T17:34:16.646 INFO:teuthology.orchestra.run.smithi110.stdout:(128/128): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:16 2024-05-04T17:34:16.651 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:16.651 INFO:teuthology.orchestra.run.smithi110.stdout:Total 6.9 MB/s | 201 MB 00:29 2024-05-04T17:34:16.685 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 73/130 2024-05-04T17:34:17.432 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/130 2024-05-04T17:34:17.478 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-05-04T17:34:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 75/130 2024-05-04T17:34:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 76/130 2024-05-04T17:34:17.580 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-05-04T17:34:17.580 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-05-04T17:34:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 77/130 2024-05-04T17:34:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 78/130 2024-05-04T17:34:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 79/130 2024-05-04T17:34:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 80/130 2024-05-04T17:34:17.959 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 81/130 2024-05-04T17:34:18.624 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 82/130 2024-05-04T17:34:18.757 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 83/130 2024-05-04T17:34:19.104 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-05-04T17:34:19.104 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-05-04T17:34:20.019 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/130 2024-05-04T17:34:20.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 85/130 2024-05-04T17:34:20.417 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 86/130 2024-05-04T17:34:20.468 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 87/130 2024-05-04T17:34:20.558 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 87/130 2024-05-04T17:34:20.589 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 88/130 2024-05-04T17:34:20.625 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 89/130 2024-05-04T17:34:21.096 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 90/130 2024-05-04T17:34:21.129 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 91/130 2024-05-04T17:34:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 91/130 2024-05-04T17:34:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 92/130 2024-05-04T17:34:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 93/130 2024-05-04T17:34:23.485 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 93/130 2024-05-04T17:34:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 93/130 2024-05-04T17:34:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 94/130 2024-05-04T17:34:24.161 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 94/130 2024-05-04T17:34:24.162 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-04T17:34:24.162 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-04T17:34:24.162 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:24.190 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 95/130 2024-05-04T17:34:24.334 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-05-04T17:34:24.416 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/130 2024-05-04T17:34:24.442 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/130 2024-05-04T17:34:24.484 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/130 2024-05-04T17:34:24.816 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/130 2024-05-04T17:34:24.934 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 5/130 2024-05-04T17:34:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 5/130 2024-05-04T17:34:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 6/130 2024-05-04T17:34:25.152 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 6/130 2024-05-04T17:34:25.194 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 7/130 2024-05-04T17:34:25.231 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/130 2024-05-04T17:34:25.268 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/130 2024-05-04T17:34:25.296 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 10/130 2024-05-04T17:34:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 10/130 2024-05-04T17:34:25.430 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86 11/130 2024-05-04T17:34:25.493 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86 11/130 2024-05-04T17:34:25.515 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 12/130 2024-05-04T17:34:25.540 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 12/130 2024-05-04T17:34:25.690 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 13/130 2024-05-04T17:34:25.723 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 13/130 2024-05-04T17:34:25.830 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 14/130 2024-05-04T17:34:25.927 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 15/130 2024-05-04T17:34:25.964 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 16/130 2024-05-04T17:34:26.007 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 17/130 2024-05-04T17:34:26.081 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/130 2024-05-04T17:34:26.115 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/130 2024-05-04T17:34:26.158 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 20/130 2024-05-04T17:34:26.191 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 21/130 2024-05-04T17:34:26.219 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 22/130 2024-05-04T17:34:26.288 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 23/130 2024-05-04T17:34:26.341 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 24/130 2024-05-04T17:34:26.371 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/130 2024-05-04T17:34:26.407 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 26/130 2024-05-04T17:34:26.483 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 27/130 2024-05-04T17:34:26.517 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-2497.gf273489c.el9. 28/130 2024-05-04T17:34:26.552 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el 29/130 2024-05-04T17:34:26.598 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_6 30/130 2024-05-04T17:34:26.669 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 31/130 2024-05-04T17:34:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 32/130 2024-05-04T17:34:26.796 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 33/130 2024-05-04T17:34:26.930 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 34/130 2024-05-04T17:34:26.982 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 35/130 2024-05-04T17:34:27.514 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 36/130 2024-05-04T17:34:27.603 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 37/130 2024-05-04T17:34:27.631 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 38/130 2024-05-04T17:34:27.674 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 39/130 2024-05-04T17:34:27.712 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_6 40/130 2024-05-04T17:34:27.745 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 41/130 2024-05-04T17:34:27.800 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 42/130 2024-05-04T17:34:27.873 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 43/130 2024-05-04T17:34:27.904 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 44/130 2024-05-04T17:34:27.941 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 45/130 2024-05-04T17:34:27.988 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 46/130 2024-05-04T17:34:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 47/130 2024-05-04T17:34:28.075 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 48/130 2024-05-04T17:34:28.220 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 49/130 2024-05-04T17:34:28.263 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 50/130 2024-05-04T17:34:28.301 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 51/130 2024-05-04T17:34:28.396 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 52/130 2024-05-04T17:34:28.435 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 53/130 2024-05-04T17:34:28.475 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 54/130 2024-05-04T17:34:28.500 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 55/130 2024-05-04T17:34:28.525 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 56/130 2024-05-04T17:34:28.591 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 57/130 2024-05-04T17:34:28.736 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 58/130 2024-05-04T17:34:28.769 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 59/130 2024-05-04T17:34:28.805 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 60/130 2024-05-04T17:34:28.846 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : sqlite-3.34.1-7.el9.x86_64 61/130 2024-05-04T17:34:28.881 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 62/130 2024-05-04T17:34:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 63/130 2024-05-04T17:34:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 64/130 2024-05-04T17:34:28.963 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 65/130 2024-05-04T17:34:29.004 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 66/130 2024-05-04T17:34:29.039 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 67/130 2024-05-04T17:34:29.125 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 68/130 2024-05-04T17:34:29.169 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 69/130 2024-05-04T17:34:29.200 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 70/130 2024-05-04T17:34:29.248 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 71/130 2024-05-04T17:34:29.346 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 72/130 2024-05-04T17:34:29.402 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 73/130 2024-05-04T17:34:30.132 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/130 2024-05-04T17:34:30.174 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 75/130 2024-05-04T17:34:30.208 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 76/130 2024-05-04T17:34:30.308 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 77/130 2024-05-04T17:34:30.350 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 78/130 2024-05-04T17:34:30.527 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 79/130 2024-05-04T17:34:30.556 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 80/130 2024-05-04T17:34:30.633 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 81/130 2024-05-04T17:34:31.310 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 82/130 2024-05-04T17:34:31.422 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 83/130 2024-05-04T17:34:32.662 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/130 2024-05-04T17:34:32.754 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 85/130 2024-05-04T17:34:33.064 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 86/130 2024-05-04T17:34:33.100 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 87/130 2024-05-04T17:34:33.190 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 87/130 2024-05-04T17:34:33.221 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 88/130 2024-05-04T17:34:33.257 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 89/130 2024-05-04T17:34:33.725 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 90/130 2024-05-04T17:34:33.752 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 91/130 2024-05-04T17:34:33.815 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 91/130 2024-05-04T17:34:33.837 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 92/130 2024-05-04T17:34:35.400 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 95/130 2024-05-04T17:34:35.400 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-05-04T17:34:35.401 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:35.574 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 96/130 2024-05-04T17:34:35.590 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 96/130 2024-05-04T17:34:35.590 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:35.590 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-04T17:34:35.590 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:36.155 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 93/130 2024-05-04T17:34:36.194 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 93/130 2024-05-04T17:34:36.738 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 93/130 2024-05-04T17:34:36.772 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 94/130 2024-05-04T17:34:36.867 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 94/130 2024-05-04T17:34:36.867 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-04T17:34:36.867 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-04T17:34:36.867 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:36.899 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 95/130 2024-05-04T17:34:37.075 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 97/130 2024-05-04T17:34:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 97/130 2024-05-04T17:34:37.102 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:37.102 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-04T17:34:37.103 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:34:37.103 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:34:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 98/130 2024-05-04T17:34:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 98/130 2024-05-04T17:34:37.348 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:37.348 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-04T17:34:37.348 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:34:37.348 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:34:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:37.801 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 99/130 2024-05-04T17:34:37.824 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 99/130 2024-05-04T17:34:37.824 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:37.824 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-04T17:34:37.824 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:34:37.824 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:34:37.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:37.887 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 100/130 2024-05-04T17:34:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 101/130 2024-05-04T17:34:38.128 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 102/130 2024-05-04T17:34:38.175 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 103/130 2024-05-04T17:34:38.254 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 104/130 2024-05-04T17:34:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 105/130 2024-05-04T17:34:38.490 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 106/130 2024-05-04T17:34:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 107/130 2024-05-04T17:34:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 108/130 2024-05-04T17:34:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 109/130 2024-05-04T17:34:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 110/130 2024-05-04T17:34:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 110/130 2024-05-04T17:34:39.014 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.e 111/130 2024-05-04T17:34:39.076 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c. 112/130 2024-05-04T17:34:39.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noa 113/130 2024-05-04T17:34:40.149 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noa 113/130 2024-05-04T17:34:40.191 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.n 114/130 2024-05-04T17:34:41.158 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.n 114/130 2024-05-04T17:34:41.182 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf27 115/130 2024-05-04T17:34:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf27 115/130 2024-05-04T17:34:41.446 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el 116/130 2024-05-04T17:34:41.476 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 117/130 2024-05-04T17:34:41.500 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 117/130 2024-05-04T17:34:41.500 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:41.500 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-04T17:34:41.500 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:34:41.500 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:34:41.500 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:41.552 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 118/130 2024-05-04T17:34:41.563 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 118/130 2024-05-04T17:34:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 119/130 2024-05-04T17:34:42.553 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 120/130 2024-05-04T17:34:42.578 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 121/130 2024-05-04T17:34:42.601 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 121/130 2024-05-04T17:34:42.601 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:42.601 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-04T17:34:42.601 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:42.601 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:42.601 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:42.646 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2497.gf2734 122/130 2024-05-04T17:34:42.667 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2497.gf2734 122/130 2024-05-04T17:34:42.668 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:42.668 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-04T17:34:42.668 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:42.971 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 123/130 2024-05-04T17:34:42.994 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 123/130 2024-05-04T17:34:42.994 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:42.994 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-04T17:34:42.994 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:34:42.994 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:34:42.994 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:47.492 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 124/130 2024-05-04T17:34:47.532 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 125/130 2024-05-04T17:34:47.564 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 126/130 2024-05-04T17:34:47.589 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : sqlite-devel-3.34.1-7.el9.x86_64 127/130 2024-05-04T17:34:47.620 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_ 128/130 2024-05-04T17:34:47.620 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 129/130 2024-05-04T17:34:47.704 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 129/130 2024-05-04T17:34:47.704 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 130/130 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 95/130 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-05-04T17:34:48.312 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:48.472 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 96/130 2024-05-04T17:34:48.487 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 96/130 2024-05-04T17:34:48.487 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:48.487 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-04T17:34:48.487 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:49.940 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 97/130 2024-05-04T17:34:49.965 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 97/130 2024-05-04T17:34:49.965 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:49.965 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-04T17:34:49.965 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:34:49.965 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:34:49.965 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:50.200 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 98/130 2024-05-04T17:34:50.223 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 98/130 2024-05-04T17:34:50.223 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:50.223 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-04T17:34:50.223 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:34:50.223 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:34:50.223 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:50.685 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 99/130 2024-05-04T17:34:50.708 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 99/130 2024-05-04T17:34:50.708 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:50.708 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-04T17:34:50.708 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:34:50.708 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:34:50.708 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:50.779 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 100/130 2024-05-04T17:34:50.834 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 101/130 2024-05-04T17:34:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 102/130 2024-05-04T17:34:51.084 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 103/130 2024-05-04T17:34:51.153 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 104/130 2024-05-04T17:34:51.242 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 105/130 2024-05-04T17:34:51.379 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 106/130 2024-05-04T17:34:51.417 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 107/130 2024-05-04T17:34:51.450 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 108/130 2024-05-04T17:34:51.477 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 109/130 2024-05-04T17:34:51.904 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 110/130 2024-05-04T17:34:51.934 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 110/130 2024-05-04T17:34:51.965 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.e 111/130 2024-05-04T17:34:52.027 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c. 112/130 2024-05-04T17:34:52.057 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noa 113/130 2024-05-04T17:34:53.037 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 130/130 2024-05-04T17:34:53.037 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 1/130 2024-05-04T17:34:53.037 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 2/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 3/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 4/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2497.gf2734 5/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 6/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 7/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 8/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 9/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 10/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 11/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 12/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_ 13/130 2024-05-04T17:34:53.038 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 14/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 15/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_6 16/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86 17/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 18/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el 19/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-2497.gf273489c.el9. 20/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_6 21/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 22/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 23/130 2024-05-04T17:34:53.039 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 24/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 25/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 26/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 27/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c. 28/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noa 29/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.n 30/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf27 31/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el 32/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 33/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.e 34/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 35/130 2024-05-04T17:34:53.040 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 36/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 37/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 38/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 39/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 40/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 41/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 42/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 43/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 44/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 45/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 46/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 47/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 48/130 2024-05-04T17:34:53.041 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 49/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 55/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 56/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 57/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 59/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 60/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/130 2024-05-04T17:34:53.042 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/130 2024-05-04T17:34:53.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 77/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 78/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 79/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : sqlite-3.34.1-7.el9.x86_64 81/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : sqlite-devel-3.34.1-7.el9.x86_64 82/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 84/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 85/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 86/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 87/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 88/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 89/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 90/130 2024-05-04T17:34:53.044 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 91/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 92/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 93/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 94/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 95/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 96/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 97/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 98/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 99/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 100/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 101/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 102/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/130 2024-05-04T17:34:53.045 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 104/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 105/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 106/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 107/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 108/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 109/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 110/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 111/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 112/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 113/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 114/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 116/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 117/130 2024-05-04T17:34:53.046 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 118/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 119/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 120/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 121/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 122/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 123/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 124/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 125/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 126/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 127/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 128/130 2024-05-04T17:34:53.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 129/130 2024-05-04T17:34:53.080 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noa 113/130 2024-05-04T17:34:53.117 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.n 114/130 2024-05-04T17:34:54.074 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.n 114/130 2024-05-04T17:34:54.101 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf27 115/130 2024-05-04T17:34:54.201 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf27 115/130 2024-05-04T17:34:54.355 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el 116/130 2024-05-04T17:34:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 117/130 2024-05-04T17:34:54.401 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 117/130 2024-05-04T17:34:54.401 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:54.401 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-04T17:34:54.401 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:34:54.401 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:34:54.401 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:54.445 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 118/130 2024-05-04T17:34:54.456 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 118/130 2024-05-04T17:34:54.553 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 119/130 2024-05-04T17:34:54.836 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 130/130 2024-05-04T17:34:54.836 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:54.836 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-05-04T17:34:54.836 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.836 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.836 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.837 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-04T17:34:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-04T17:34:54.839 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-04T17:34:54.840 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-04T17:34:54.841 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-04T17:34:54.842 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-04T17:34:54.843 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-04T17:34:54.844 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: sqlite-3.34.1-7.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: sqlite-devel-3.34.1-7.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-04T17:34:54.845 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:34:54.846 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-05-04T17:34:55.179 DEBUG:teuthology.parallel:result is None 2024-05-04T17:34:55.442 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 120/130 2024-05-04T17:34:55.472 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 121/130 2024-05-04T17:34:55.494 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 121/130 2024-05-04T17:34:55.494 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:55.494 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-04T17:34:55.494 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:55.494 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:55.494 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:55.539 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2497.gf2734 122/130 2024-05-04T17:34:55.561 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2497.gf2734 122/130 2024-05-04T17:34:55.561 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:55.561 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-04T17:34:55.561 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:34:55.856 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 123/130 2024-05-04T17:34:55.879 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 123/130 2024-05-04T17:34:55.879 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:55.879 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-04T17:34:55.879 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:34:55.880 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:34:55.880 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:00.405 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 124/130 2024-05-04T17:35:00.455 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 125/130 2024-05-04T17:35:00.482 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 126/130 2024-05-04T17:35:00.507 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : sqlite-devel-3.34.1-7.el9.x86_64 127/130 2024-05-04T17:35:00.522 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_ 128/130 2024-05-04T17:35:00.522 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 129/130 2024-05-04T17:35:00.606 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 129/130 2024-05-04T17:35:00.606 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 130/130 2024-05-04T17:35:05.875 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 130/130 2024-05-04T17:35:05.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 1/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 2/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 3/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 4/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2497.gf2734 5/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 6/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 7/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 8/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 9/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 10/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 11/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 12/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_ 13/130 2024-05-04T17:35:05.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 14/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 15/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_6 16/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86 17/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 18/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el 19/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-2497.gf273489c.el9. 20/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_6 21/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 22/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 23/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 24/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 25/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 26/130 2024-05-04T17:35:05.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 27/130 2024-05-04T17:35:05.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c. 28/130 2024-05-04T17:35:05.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noa 29/130 2024-05-04T17:35:05.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.n 30/130 2024-05-04T17:35:05.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf27 31/130 2024-05-04T17:35:05.887 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el 32/130 2024-05-04T17:35:05.887 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 33/130 2024-05-04T17:35:05.887 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.e 34/130 2024-05-04T17:35:05.887 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 35/130 2024-05-04T17:35:05.887 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 36/130 2024-05-04T17:35:05.887 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 37/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 38/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 39/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 40/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 41/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 42/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 43/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 44/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 45/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 46/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 47/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 48/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 49/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/130 2024-05-04T17:35:05.888 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 55/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 56/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 57/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 59/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 60/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/130 2024-05-04T17:35:05.889 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 77/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 78/130 2024-05-04T17:35:05.890 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 79/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : sqlite-3.34.1-7.el9.x86_64 81/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : sqlite-devel-3.34.1-7.el9.x86_64 82/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 84/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 85/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 86/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 87/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 88/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 89/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 90/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 91/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 92/130 2024-05-04T17:35:05.891 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 93/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 94/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 95/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 96/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 97/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 98/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 99/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 100/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 101/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 102/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 104/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 105/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 106/130 2024-05-04T17:35:05.892 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 107/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 108/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 109/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 110/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 111/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 112/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 113/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 114/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 116/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 117/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 118/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 119/130 2024-05-04T17:35:05.893 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 120/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 121/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 122/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 123/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 124/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 125/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 126/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 127/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 128/130 2024-05-04T17:35:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 129/130 2024-05-04T17:35:07.788 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 130/130 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-04T17:35:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.793 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-04T17:35:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-04T17:35:07.795 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-04T17:35:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-04T17:35:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-05-04T17:35:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-04T17:35:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: sqlite-3.34.1-7.el9.x86_64 2024-05-04T17:35:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: sqlite-devel-3.34.1-7.el9.x86_64 2024-05-04T17:35:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-04T17:35:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-04T17:35:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:07.799 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-05-04T17:35:08.149 DEBUG:teuthology.parallel:result is None 2024-05-04T17:35:08.149 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:35:08.295 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-04T17:35:08.316 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-2497.gf273489c.el9 2024-05-04T17:35:08.316 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2497.gf273489c.el9 2024-05-04T17:35:08.317 INFO:teuthology.task.install:The correct ceph version 19.0.0-2497.gf273489c is installed. 2024-05-04T17:35:08.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:35:08.471 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-04T17:35:08.497 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-2497.gf273489c.el9 2024-05-04T17:35:08.497 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2497.gf273489c.el9 2024-05-04T17:35:08.497 INFO:teuthology.task.install:The correct ceph version 19.0.0-2497.gf273489c is installed. 2024-05-04T17:35:08.499 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-04T17:35:08.499 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:08.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-04T17:35:08.525 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:08.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-04T17:35:08.567 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-04T17:35:08.567 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:08.567 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-05-04T17:35:08.591 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-04T17:35:08.654 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:08.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-05-04T17:35:08.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-04T17:35:08.744 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-04T17:35:08.745 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:08.745 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-04T17:35:08.769 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-04T17:35:08.833 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:08.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-04T17:35:08.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-04T17:35:08.922 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-04T17:35:08.922 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:08.922 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-05-04T17:35:08.946 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-04T17:35:09.009 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:09.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-05-04T17:35:09.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-04T17:35:09.096 INFO:teuthology.run_tasks:Running task ceph... 2024-05-04T17:35:09.185 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-04T17:35:09.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-05-04T17:35:09.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-05-04T17:35:09.211 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-04T17:35:09.211 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-04T17:35:09.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-04T17:35:09.274 INFO:tasks.ceph:Creating extra log directories... 2024-05-04T17:35:09.274 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-04T17:35:09.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-04T17:35:09.349 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-04T17:35:09.349 INFO:tasks.ceph:config {'conf': {'client': {'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 1500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-05-04T17:35:09.349 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130', 'branch': 'wip-yuriw11-testing-20240501.200505-squid', 'description': 'rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7687130', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuriw11-testing-20240501.200505-squid'}, 'ceph': {'conf': {'client': {'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 1500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled'], 'mon_bind_msgr2': False, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw11-testing-20240501.200505-squid', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8797, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuriw11-testing-20240501.200505-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCnwtQ3UjX+ovj6EqFBJ7I+26RDiS5MVlG5G0jhIn8se7gSyirglsz7eaEGZBORk40HF4wjMixPVeqB7c5I2CY=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8t/NqQ6azIROAkQMLsXeHZC4bcS+tYulmkYG2am70Vu8aF97HazWAv0e+WT8ymGt3Xz3hNSBcMUs36kwJFufQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}}, {'ceph': None}, {'exec': {'client.0': ['ceph osd pool create cephsqlite', "ceph auth get-or-create client.libcephsqlite mon 'profile simple-rados-client-with-blocklist' osd 'allow rwx pool=cephsqlite' >> /etc/ceph/ceph.keyring"]}}, {'exec': {'client.0': ['ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr']}}, {'workunit': {'clients': {'client.0': ['rados/test_libcephsqlite.sh cephsqlite']}, 'env': {'CEPH_ARGS': '--id libcephsqlite --no-log-to-stderr'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-02_23:59:28', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-04T17:35:09.350 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-04T17:35:09.377 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-04T17:35:09.393 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-05-04T17:35:09.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-05-04T17:35:09.456 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:09.457 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-05-04T17:35:09.508 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-04T17:35:09.509 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-05-04T17:35:09.563 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-04 17:35:01.128605268 +0000 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-04 17:33:04.617793892 +0000 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-04 17:33:04.617793892 +0000 2024-05-04T17:35:09.564 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-04 17:33:04.617793892 +0000 2024-05-04T17:35:09.564 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-04T17:35:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-04T17:35:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-04T17:35:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.4345e-05 s, 6.1 MB/s 2024-05-04T17:35:09.626 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-04T17:35:09.683 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-05-04T17:35:09.737 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-04T17:35:09.737 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:09.737 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-05-04T17:35:09.737 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:09.738 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:09.738 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-04 17:35:01.128605268 +0000 2024-05-04T17:35:09.738 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-04 17:33:04.933788231 +0000 2024-05-04T17:35:09.738 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-04 17:33:04.933788231 +0000 2024-05-04T17:35:09.738 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-04 17:33:04.933788231 +0000 2024-05-04T17:35:09.738 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-04T17:35:09.800 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-04T17:35:09.800 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-04T17:35:09.800 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000114158 s, 4.5 MB/s 2024-05-04T17:35:09.801 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-04T17:35:09.856 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-05-04T17:35:09.912 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-04T17:35:09.912 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:09.912 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-04T17:35:09.912 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:09.913 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:09.913 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-04 17:35:01.129605251 +0000 2024-05-04T17:35:09.913 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-04 17:33:05.263782319 +0000 2024-05-04T17:35:09.913 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-04 17:33:05.263782319 +0000 2024-05-04T17:35:09.913 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-04 17:33:05.263782319 +0000 2024-05-04T17:35:09.913 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-04T17:35:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-04T17:35:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-04T17:35:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.9895e-05 s, 5.1 MB/s 2024-05-04T17:35:09.975 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-04T17:35:10.031 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-05-04T17:35:10.087 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-04T17:35:10.087 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:10.087 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-04T17:35:10.088 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:10.088 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:10.088 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-04 17:35:01.129605251 +0000 2024-05-04T17:35:10.088 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-04 17:33:05.754773524 +0000 2024-05-04T17:35:10.088 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-04 17:33:05.754773524 +0000 2024-05-04T17:35:10.088 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-04 17:33:05.754773524 +0000 2024-05-04T17:35:10.088 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-04T17:35:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-04T17:35:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-04T17:35:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.5774e-05 s, 6.8 MB/s 2024-05-04T17:35:10.151 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-04T17:35:10.206 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:10.207 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:10.207 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-05-04T17:35:10.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-04T17:35:10.223 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-05-04T17:35:10.278 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-04T17:35:10.278 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-04 17:34:48.172146610 +0000 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-04 17:33:04.633222288 +0000 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-04 17:33:04.633222288 +0000 2024-05-04T17:35:10.279 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-05-04 17:33:04.633222288 +0000 2024-05-04T17:35:10.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-04T17:35:10.340 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-04T17:35:10.341 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-04T17:35:10.341 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000185643 s, 2.8 MB/s 2024-05-04T17:35:10.342 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-04T17:35:10.397 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-05-04T17:35:10.452 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-04T17:35:10.452 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-04 17:34:48.172146610 +0000 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-04 17:33:04.974220998 +0000 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-04 17:33:04.974220998 +0000 2024-05-04T17:35:10.453 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-05-04 17:33:04.974220998 +0000 2024-05-04T17:35:10.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-04T17:35:10.515 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-04T17:35:10.515 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-04T17:35:10.515 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00020563 s, 2.5 MB/s 2024-05-04T17:35:10.516 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-04T17:35:10.573 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-05-04T17:35:10.627 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-04T17:35:10.627 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:10.627 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-04T17:35:10.627 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:10.628 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:10.628 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-04 17:34:48.173146605 +0000 2024-05-04T17:35:10.628 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-04 17:33:05.326219666 +0000 2024-05-04T17:35:10.628 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-04 17:33:05.326219666 +0000 2024-05-04T17:35:10.628 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-05-04 17:33:05.326219666 +0000 2024-05-04T17:35:10.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-04T17:35:10.689 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-04T17:35:10.689 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-04T17:35:10.690 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000182494 s, 2.8 MB/s 2024-05-04T17:35:10.691 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-04T17:35:10.746 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-05-04T17:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-04T17:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-04T17:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-04 17:34:48.173146605 +0000 2024-05-04T17:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-04 17:33:05.680218327 +0000 2024-05-04T17:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-04 17:33:05.680218327 +0000 2024-05-04T17:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-05-04 17:33:05.680218327 +0000 2024-05-04T17:35:10.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-04T17:35:10.863 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-04T17:35:10.863 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-04T17:35:10.863 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000167621 s, 3.1 MB/s 2024-05-04T17:35:10.865 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-04T17:35:10.920 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:10.921 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-04T17:35:10.921 INFO:tasks.ceph:Generating config... 2024-05-04T17:35:10.922 INFO:tasks.ceph:[client] debug cephsqlite = 20 2024-05-04T17:35:10.922 INFO:tasks.ceph:[client] debug client = 20 2024-05-04T17:35:10.922 INFO:tasks.ceph:[client] debug ms = 1 2024-05-04T17:35:10.922 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-04T17:35:10.922 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-05-04T17:35:10.922 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-05-04T17:35:10.922 INFO:tasks.ceph:[global] ms inject socket failures = 1500 2024-05-04T17:35:10.923 INFO:tasks.ceph:[global] ms type = async 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-04T17:35:10.923 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-05-04T17:35:10.923 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-04T17:35:10.923 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-04T17:35:10.923 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-04T17:35:10.924 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-04T17:35:10.925 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-04T17:35:10.925 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-04T17:35:10.925 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-04T17:35:10.925 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-04T17:35:10.925 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-04T17:35:10.925 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-04T17:35:10.925 INFO:tasks.ceph:Setting up mon.a... 2024-05-04T17:35:10.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-04T17:35:10.976 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2024-05-04T17:35:10.979 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-04T17:35:11.022 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-04T17:35:11.087 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.110:6789]'), ('mon.c', '[v1:172.21.15.110:6790]'), ('mon.b', '[v1:172.21.15.136:6789]')] 2024-05-04T17:35:11.087 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.110:6789],[v1:172.21.15.110:6790],[v1:172.21.15.136:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 1500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-04T17:35:11.088 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:11.088 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-04T17:35:11.142 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.110:6789]' --addv c '[v1:172.21.15.110:6790]' --addv b '[v1:172.21.15.136:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:11.228 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:11.228 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 5cbb7704-6b22-42de-b82d-75ed642c3bca 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = quincy 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 5cbb7704-6b22-42de-b82d-75ed642c3bca 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2024-05-04T17:35:11.228722+0000 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:created 2024-05-04T17:35:11.228722+0000 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 17 (quincy) 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:0: v1:172.21.15.110:6789/0 mon.a 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:1: v1:172.21.15.136:6789/0 mon.b 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:2: v1:172.21.15.110:6790/0 mon.c 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-04T17:35:11.231 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-04T17:35:11.245 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5cbb7704-6b22-42de-b82d-75ed642c3bca... 2024-05-04T17:35:11.246 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-04T17:35:11.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-04T17:35:11.324 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.325 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-05-04T17:35:11.326 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.327 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "[v1:172.21.15.110:6789],[v1:172.21.15.110:6790],[v1:172.21.15.136:6789]" 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr2 = False 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 1500 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 5cbb7704-6b22-42de-b82d-75ed642c3bca 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.328 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-05-04T17:35:11.329 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-05-04T17:35:11.330 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = stupid 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 20 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 20 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-05-04T17:35:11.331 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-05-04T17:35:11.332 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.333 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-05-04T17:35:11.334 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2024-05-04T17:35:11.335 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: debug cephsqlite = 20 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: debug client = 20 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-05-04T17:35:11.336 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-05-04T17:35:11.337 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.338 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-04T17:35:11.339 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-05-04T17:35:11.340 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "[v1:172.21.15.110:6789],[v1:172.21.15.110:6790],[v1:172.21.15.136:6789]" 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: mon election default strategy = 1 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: ms bind msgr2 = False 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 1500 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: ms type = async 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 5cbb7704-6b22-42de-b82d-75ed642c3bca 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-05-04T17:35:11.341 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.342 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.343 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: bdev async discard = True 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: bdev enable discard = True 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore allocator = stupid 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore zero block detection = True 2024-05-04T17:35:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 20 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 20 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 10 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify cached snaps = True 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify missing on start = True 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock override recovery settings = True 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue = debug_random 2024-05-04T17:35:11.345 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue cut off = debug_random 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: osd shutdown pgref assert = True 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-05-04T17:35:11.346 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.347 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: mon min osdmap epochs = 50 2024-05-04T17:35:11.348 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune interval = 2 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune min = 15 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune txsize = 2 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub interval = 300 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: paxos service trim min = 10 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-04T17:35:11.349 INFO:teuthology.orchestra.run.smithi136.stdout: debug cephsqlite = 20 2024-05-04T17:35:11.350 INFO:teuthology.orchestra.run.smithi136.stdout: debug client = 20 2024-05-04T17:35:11.350 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-04T17:35:11.350 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-05-04T17:35:11.350 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2024-05-04T17:35:11.350 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2024-05-04T17:35:11.352 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-04T17:35:11.352 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-04T17:35:11.444 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-04T17:35:11.444 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:11.444 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-04T17:35:11.459 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:11.459 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-04T17:35:11.516 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:35:11.516 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:11.516 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-04T17:35:11.516 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-04T17:35:11.588 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:11.589 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:11.644 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:35:11.644 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:11.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-04T17:35:11.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-04T17:35:11.678 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:11.678 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:11.734 INFO:tasks.ceph:Setting up mon nodes... 2024-05-04T17:35:11.734 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-04T17:35:11.735 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-04T17:35:11.782 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-04T17:35:11.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-04T17:35:11.842 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-04T17:35:11.844 INFO:tasks.ceph:Setting up mds nodes... 2024-05-04T17:35:11.845 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-04T17:35:11.845 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-04T17:35:11.884 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-04T17:35:11.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-04T17:35:11.936 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-04T17:35:11.948 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-04T17:35:11.948 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-04T17:35:11.948 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-04T17:35:11.972 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:11.972 INFO:tasks.ceph:role: osd.0 2024-05-04T17:35:11.972 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:35:11.972 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:12.035 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:12.036 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:12.036 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:12.067 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-05-04T17:35:12.069 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-05-04T17:35:12.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-04T17:35:12.137 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-04T17:35:12.205 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-04T17:35:12.269 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:12.269 INFO:tasks.ceph:role: osd.1 2024-05-04T17:35:12.269 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:35:12.269 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-04T17:35:12.332 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:12.332 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:12.332 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:12.332 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:12.332 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:12.333 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:12.333 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:12.333 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:12.333 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:12.333 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:12.365 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-05-04T17:35:12.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-05-04T17:35:12.366 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-04T17:35:12.433 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-04T17:35:12.501 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-04T17:35:12.564 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:12.564 INFO:tasks.ceph:role: osd.2 2024-05-04T17:35:12.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:35:12.565 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:12.628 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:12.660 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-05-04T17:35:12.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-05-04T17:35:12.662 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-04T17:35:12.734 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-04T17:35:12.802 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-04T17:35:12.866 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:12.866 INFO:tasks.ceph:role: osd.3 2024-05-04T17:35:12.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:35:12.866 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:12.929 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:12.930 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:12.930 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:12.930 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:12.961 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-05-04T17:35:12.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-05-04T17:35:12.963 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-04T17:35:13.030 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-04T17:35:13.097 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:13.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:13.184+0000 7f98edd6a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-04T17:35:13.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:13.185+0000 7f98edd6a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-04T17:35:13.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:13.185+0000 7f98edd6a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-04T17:35:13.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:13.185+0000 7f98edd6a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-04T17:35:13.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:13.185+0000 7f98edd6a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-04T17:35:13.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:13.185+0000 7f98edd6a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-04T17:35:16.032 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-04T17:35:16.060 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:16.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:16.146+0000 7fb1f4a0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-04T17:35:16.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:16.146+0000 7fb1f4a0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-04T17:35:16.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:16.146+0000 7fb1f4a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-04T17:35:16.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:16.146+0000 7fb1f4a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-04T17:35:16.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:16.146+0000 7fb1f4a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-04T17:35:16.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:16.147+0000 7fb1f4a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-04T17:35:18.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-04T17:35:19.021 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:19.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:19.108+0000 7fed6a818640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-04T17:35:19.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:19.108+0000 7fed6a818640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-04T17:35:19.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:19.108+0000 7fed6a818640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-04T17:35:19.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:19.108+0000 7fed6a818640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-04T17:35:19.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:19.108+0000 7fed6a818640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-04T17:35:19.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:19.109+0000 7fed6a818640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-04T17:35:21.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-04T17:35:21.987 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:22.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:22.075+0000 7f919842e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-04T17:35:22.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:22.075+0000 7f919842e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-04T17:35:22.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:22.075+0000 7f919842e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-04T17:35:22.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:22.075+0000 7f919842e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-04T17:35:22.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:22.075+0000 7f919842e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-04T17:35:22.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:22.075+0000 7f919842e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-04T17:35:24.920 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-04T17:35:24.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-04T17:35:24.977 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:24.977 INFO:tasks.ceph:role: osd.4 2024-05-04T17:35:24.977 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:35:24.977 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:25.052 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:25.053 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:25.053 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:25.053 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:25.565 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-04T17:35:25.567 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-04T17:35:25.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-04T17:35:25.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-04T17:35:25.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-04T17:35:25.759 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:25.759 INFO:tasks.ceph:role: osd.5 2024-05-04T17:35:25.759 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:35:25.760 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-04T17:35:25.831 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:25.831 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:25.832 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:26.248 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-04T17:35:26.250 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-04T17:35:26.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-04T17:35:26.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-04T17:35:26.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-04T17:35:26.461 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:26.461 INFO:tasks.ceph:role: osd.6 2024-05-04T17:35:26.461 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:35:26.461 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-04T17:35:26.534 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:26.535 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:26.535 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:26.535 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:26.535 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:26.536 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:26.536 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:26.536 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:26.536 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:26.536 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:27.056 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-04T17:35:27.058 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-04T17:35:27.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-04T17:35:27.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-04T17:35:27.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-04T17:35:27.260 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:35:27.260 INFO:tasks.ceph:role: osd.7 2024-05-04T17:35:27.261 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:35:27.261 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-04T17:35:27.334 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:35:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:35:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:35:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:35:27.335 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:35:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:35:27.335 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:35:27.335 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:35:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:35:27.335 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:35:27.853 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-04T17:35:27.855 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-04T17:35:27.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-04T17:35:27.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-04T17:35:27.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:28.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:28.088+0000 7fbc03210640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-04T17:35:28.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:28.089+0000 7fbc03210640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-04T17:35:28.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:28.089+0000 7fbc03210640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-04T17:35:28.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:28.089+0000 7fbc03210640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-04T17:35:28.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:28.089+0000 7fbc03210640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-04T17:35:28.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:28.090+0000 7fbc03210640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-04T17:35:30.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-04T17:35:30.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:31.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:31.062+0000 7f999e40d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-04T17:35:31.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:31.062+0000 7f999e40d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-04T17:35:31.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:31.062+0000 7f999e40d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-04T17:35:31.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:31.062+0000 7f999e40d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-04T17:35:31.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:31.062+0000 7f999e40d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-04T17:35:31.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:31.063+0000 7f999e40d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-04T17:35:33.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-04T17:35:33.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:34.041+0000 7f24bed06640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-04T17:35:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:34.042+0000 7f24bed06640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-04T17:35:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:34.042+0000 7f24bed06640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-04T17:35:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:34.042+0000 7f24bed06640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-04T17:35:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:34.042+0000 7f24bed06640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-04T17:35:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:34.042+0000 7f24bed06640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-04T17:35:36.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-04T17:35:36.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:37.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:37.008+0000 7fe853601640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-04T17:35:37.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:37.009+0000 7fe853601640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-04T17:35:37.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:37.009+0000 7fe853601640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-04T17:35:37.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:37.009+0000 7fe853601640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-04T17:35:37.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:37.009+0000 7fe853601640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-04T17:35:37.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:37.009+0000 7fe853601640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-04T17:35:39.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-04T17:35:39.885 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-04T17:35:39.886 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:39.886 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-04T17:35:39.915 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:39.915 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-04T17:35:39.982 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:39.982 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-04T17:35:40.048 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:40.048 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-04T17:35:40.114 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:40.114 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-04T17:35:40.180 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:40.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-04T17:35:40.208 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:40.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-04T17:35:40.275 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:40.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-04T17:35:40.341 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:40.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-04T17:35:40.407 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:40.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-04T17:35:40.472 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:40.472 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-04T17:35:40.488 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:40.488 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-04T17:35:40.528 INFO:tasks.ceph:Adding keys to all mons... 2024-05-04T17:35:40.528 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-04T17:35:40.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-04T17:35:40.572 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2024-05-04T17:35:40.572 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDPcTZmXJCsLhAA8VbFGN7tRKkUMm4Ax/yziA== 2024-05-04T17:35:40.572 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDRcTZmXw4ZCxAAtS1Pyt4Cvmdefz1MnpZlMA== 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDUcTZm6ZbPCBAAC4TKcY5yUG4uOuMBfSCLkg== 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDXcTZmWXGMBhAAjqc0QlqT+9K06U4MpWd8vA== 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDacTZm2f2KBBAAfQp9yAKFhXryqato9ChBvA== 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDPcTZma3UzMhAATFag62ewFQ/nc5hoRuYrUQ== 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-05-04T17:35:40.573 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDgcTZmsNdnBRAAdhlebow5HuArixlRev23Bw== 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDjcTZmUi/UAxAA/2DVz5EmzQjjfsOgN/Tzng== 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDmcTZm9MqNAhAAeIi2DvhNPUCUXai3Zwo4pg== 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDpcTZmDBmVABAAR2h5qGwhkoUd5uygHEcIMA== 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDPcTZmKua0NBAAP1yUjNSc2dlZcSglk8lY+Q== 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2024-05-04T17:35:40.574 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDPcTZmW8rWNxAAJBXJcB2UaAZqoG7J1vcIsQ== 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDPcTZmXJCsLhAA8VbFGN7tRKkUMm4Ax/yziA== 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDRcTZmXw4ZCxAAtS1Pyt4Cvmdefz1MnpZlMA== 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDUcTZm6ZbPCBAAC4TKcY5yUG4uOuMBfSCLkg== 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDXcTZmWXGMBhAAjqc0QlqT+9K06U4MpWd8vA== 2024-05-04T17:35:40.591 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDacTZm2f2KBBAAfQp9yAKFhXryqato9ChBvA== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDPcTZma3UzMhAATFag62ewFQ/nc5hoRuYrUQ== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDgcTZmsNdnBRAAdhlebow5HuArixlRev23Bw== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDjcTZmUi/UAxAA/2DVz5EmzQjjfsOgN/Tzng== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDmcTZm9MqNAhAAeIi2DvhNPUCUXai3Zwo4pg== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDpcTZmDBmVABAAR2h5qGwhkoUd5uygHEcIMA== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDPcTZmKua0NBAAP1yUjNSc2dlZcSglk8lY+Q== 2024-05-04T17:35:40.592 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-05-04T17:35:40.593 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDPcTZmW8rWNxAAJBXJcB2UaAZqoG7J1vcIsQ== 2024-05-04T17:35:40.594 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:35:40.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:35:40.683 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.734 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.785 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.838 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.888 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:35:40.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:35:40.939 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.989 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:40.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:41.040 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:41.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:41.091 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:41.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:35:41.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:35:41.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:35:41.195 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:35:41.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:35:41.249 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-04T17:35:41.249 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-04T17:35:41.273 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-04T17:35:41.366 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-04T17:35:41.390 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-04T17:35:41.453 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-04T17:35:41.547 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-04T17:35:41.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-04T17:35:41.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-04T17:35:41.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-04T17:35:41.711 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:41.713 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:35:41.766 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-04T17:35:41.767 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-04T17:35:41.767 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-04T17:35:41.769 INFO:tasks.ceph.mon.a:Started 2024-05-04T17:35:41.769 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-04T17:35:41.769 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-04T17:35:41.771 INFO:tasks.ceph.mon.c:Started 2024-05-04T17:35:41.771 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-04T17:35:41.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-04T17:35:41.809 INFO:tasks.ceph.mon.b:Started 2024-05-04T17:35:41.809 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-04T17:35:41.810 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-04T17:35:41.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-04T17:35:41.812 INFO:tasks.ceph.mgr.y:Started 2024-05-04T17:35:41.812 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-04T17:35:41.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-04T17:35:41.814 INFO:tasks.ceph.mgr.x:Started 2024-05-04T17:35:41.814 INFO:tasks.ceph:Setting crush tunables to default 2024-05-04T17:35:41.815 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2024-05-04T17:35:41.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.912+0000 7f1dec196640 1 Processor -- start 2024-05-04T17:35:41.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.912+0000 7f1dec196640 1 -- start start 2024-05-04T17:35:41.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.913+0000 7f1dec196640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de40afea0 con 0x7f1de40abe00 2024-05-04T17:35:41.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.913+0000 7f1dec196640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de4072660 con 0x7f1de413c260 2024-05-04T17:35:41.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.913+0000 7f1dec196640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de413c670 con 0x7f1de40ae6e0 2024-05-04T17:35:41.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.913+0000 7f1de9f32640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f1de40ae6e0 0x7f1de40aad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:58854/0 (socket says 172.21.15.110:58854) 2024-05-04T17:35:41.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.913+0000 7f1de9731640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1de40abe00 0x7f1de40ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33026/0 (socket says 172.21.15.110:33026) 2024-05-04T17:35:41.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:41.913+0000 7f1de9f32640 1 -- 172.21.15.110:0/3752282539 learned_addr learned my addr 172.21.15.110:0/3752282539 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:44.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:44.913+0000 7f1ddb7fe640 1 -- 172.21.15.110:0/3752282539 >> v1:172.21.15.110:6790/0 conn(0x7f1de40ae6e0 legacy=0x7f1de40aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:44.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:44.913+0000 7f1ddb7fe640 1 -- 172.21.15.110:0/3752282539 >> v1:172.21.15.136:6789/0 conn(0x7f1de413c260 legacy=0x7f1de414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:44.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:44.913+0000 7f1ddb7fe640 1 -- 172.21.15.110:0/3752282539 >> v1:172.21.15.110:6789/0 conn(0x7f1de40abe00 legacy=0x7f1de40ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:44.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:44.913+0000 7f1ddb7fe640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de413c840 con 0x7f1de40abe00 2024-05-04T17:35:44.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:44.913+0000 7f1ddb7fe640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc8011a80 con 0x7f1dc8009580 2024-05-04T17:35:44.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:44.913+0000 7f1ddb7fe640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc8012cd0 con 0x7f1dc80058f0 2024-05-04T17:35:47.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.076+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056770188 0 0) 0x7f1de413c840 con 0x7f1de40abe00 2024-05-04T17:35:47.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.076+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dd0003520 con 0x7f1de40abe00 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3949618898 0 0) 0x7f1dc8012cd0 con 0x7f1dc80058f0 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1de413c840 con 0x7f1dc80058f0 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728808649 0 0) 0x7f1dc8011a80 con 0x7f1dc8009580 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dc8012cd0 con 0x7f1dc8009580 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1094949922 0 0) 0x7f1dd0003520 con 0x7f1de40abe00 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 >> v1:172.21.15.110:6790/0 conn(0x7f1dc80058f0 legacy=0x7f1dc8007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 >> v1:172.21.15.136:6789/0 conn(0x7f1dc8009580 legacy=0x7f1dc800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc8013f60 con 0x7f1de40abe00 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ddc001a10 con 0x7f1de40abe00 2024-05-04T17:35:47.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ddc001c70 con 0x7f1de40abe00 2024-05-04T17:35:47.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.077+0000 7f1de8f30640 1 -- 172.21.15.110:0/3752282539 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ddc002080 con 0x7f1de40abe00 2024-05-04T17:35:47.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.078+0000 7f1dec196640 1 -- 172.21.15.110:0/3752282539 >> v1:172.21.15.110:6789/0 conn(0x7f1de40abe00 legacy=0x7f1dc800f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.078+0000 7f1dec196640 1 -- 172.21.15.110:0/3752282539 shutdown_connections 2024-05-04T17:35:47.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.078+0000 7f1dec196640 1 -- 172.21.15.110:0/3752282539 >> 172.21.15.110:0/3752282539 conn(0x7f1de40b3c50 msgr2=0x7f1de40b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:47.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.079+0000 7f1dec196640 1 -- 172.21.15.110:0/3752282539 shutdown_connections 2024-05-04T17:35:47.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.079+0000 7f1dec196640 1 -- 172.21.15.110:0/3752282539 wait complete. 2024-05-04T17:35:47.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.080+0000 7f1dec196640 1 Processor -- start 2024-05-04T17:35:47.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.080+0000 7f1dec196640 1 -- start start 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.081+0000 7f1dec196640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de413fd30 con 0x7f1dc8009580 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.081+0000 7f1dec196640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de40717b0 con 0x7f1de40abe00 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.081+0000 7f1dec196640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de4071980 con 0x7f1dc80058f0 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.081+0000 7f1de9731640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1dc8009580 0x7f1de413da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37598/0 (socket says 172.21.15.110:37598) 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.081+0000 7f1de9731640 1 -- 172.21.15.110:0/3775721165 learned_addr learned my addr 172.21.15.110:0/3775721165 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107948009 0 0) 0x7f1de40717b0 con 0x7f1de40abe00 2024-05-04T17:35:47.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dc800c2a0 con 0x7f1de40abe00 2024-05-04T17:35:47.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 16072505 0 0) 0x7f1dc800c2a0 con 0x7f1de40abe00 2024-05-04T17:35:47.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1de40717b0 con 0x7f1de40abe00 2024-05-04T17:35:47.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ddc001910 con 0x7f1de40abe00 2024-05-04T17:35:47.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723701441 0 0) 0x7f1de413fd30 con 0x7f1dc8009580 2024-05-04T17:35:47.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.082+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dc800c2a0 con 0x7f1dc8009580 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385498300 0 0) 0x7f1de4071980 con 0x7f1dc80058f0 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1de413fd30 con 0x7f1dc80058f0 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2692204271 0 0) 0x7f1de40717b0 con 0x7f1de40abe00 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 >> v1:172.21.15.110:6790/0 conn(0x7f1dc80058f0 legacy=0x7f1de41dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 >> v1:172.21.15.110:6789/0 conn(0x7f1dc8009580 legacy=0x7f1de413da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de41e60f0 con 0x7f1de40abe00 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1de41e30c0 con 0x7f1de40abe00 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1de41e3590 con 0x7f1de40abe00 2024-05-04T17:35:47.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ddc003c20 con 0x7f1de40abe00 2024-05-04T17:35:47.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ddc003f70 con 0x7f1de40abe00 2024-05-04T17:35:47.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.084+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1ddc0044a0 con 0x7f1de40abe00 2024-05-04T17:35:47.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.085+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1252519728 0 0) 0x7f1ddc005b00 con 0x7f1de40abe00 2024-05-04T17:35:47.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.086+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de41db4f0 con 0x7f1de40abe00 2024-05-04T17:35:47.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.088+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1ddc004920 con 0x7f1de40abe00 2024-05-04T17:35:47.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.159+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f1de404fe40 con 0x7f1de40abe00 2024-05-04T17:35:47.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.164+0000 7f1dda7fc640 1 -- 172.21.15.110:0/3775721165 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f1ddc004b80 con 0x7f1de40abe00 2024-05-04T17:35:47.165 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2024-05-04T17:35:47.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.165+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 >> v1:172.21.15.136:6789/0 conn(0x7f1de40abe00 legacy=0x7f1de413e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.165+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 shutdown_connections 2024-05-04T17:35:47.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.165+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 >> 172.21.15.110:0/3775721165 conn(0x7f1de40b3c50 msgr2=0x7f1de4134b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:47.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.165+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 shutdown_connections 2024-05-04T17:35:47.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:47.165+0000 7f1dec196640 1 -- 172.21.15.110:0/3775721165 wait complete. 2024-05-04T17:35:47.177 INFO:tasks.ceph:check_enable_crimson: False 2024-05-04T17:35:47.177 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-04T17:35:47.177 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:47.177 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-04T17:35:47.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:47.198+0000 7f3cc3014040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-04T17:35:47.201 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:47.202 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-04T17:35:47.213 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:47.212+0000 7f28c840f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-04T17:35:47.265 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:47.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-04T17:35:47.328 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:35:47.329 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-04T17:35:47.355 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:47.354+0000 7f28c840f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:35:47.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:47.355+0000 7f3cc3014040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:35:47.379 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:47.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-04T17:35:47.402 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:47.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-04T17:35:47.410 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:47.409+0000 7f28c840f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-04T17:35:47.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:47.411+0000 7f3cc3014040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-04T17:35:47.432 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:47.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-04T17:35:47.496 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-04T17:35:47.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-04T17:35:47.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ac107878-2a9c-435d-a24b-634939607eb5 0 2024-05-04T17:35:47.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:47.649+0000 7f3cc3014040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-04T17:35:47.675 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:47.674+0000 7f28c840f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-04T17:35:47.695 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:47.694+0000 7f3cc3014040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-04T17:35:47.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.695+0000 7f4381721640 1 Processor -- start 2024-05-04T17:35:47.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.696+0000 7f4381721640 1 -- start start 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.696+0000 7f4381721640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c0ae510 con 0x7f437c06ba20 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.696+0000 7f4381721640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c145960 con 0x7f437c06cb80 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.696+0000 7f4381721640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c13fe50 con 0x7f437c0ae930 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.696+0000 7f437a7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f437c06cb80 0x7f437c145e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52758/0 (socket says 172.21.15.136:52758) 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.696+0000 7f437a7fc640 1 -- 172.21.15.136:0/3690954200 learned_addr learned my addr 172.21.15.136:0/3690954200 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 443586366 0 0) 0x7f437c145960 con 0x7f437c06cb80 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4364003520 con 0x7f437c06cb80 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2538029531 0 0) 0x7f437c0ae510 con 0x7f437c06ba20 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f437c145960 con 0x7f437c06ba20 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316452796 0 0) 0x7f437c13fe50 con 0x7f437c0ae930 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f437c0ae510 con 0x7f437c0ae930 2024-05-04T17:35:47.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3945342588 0 0) 0x7f4364003520 con 0x7f437c06cb80 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 >> v1:172.21.15.110:6790/0 conn(0x7f437c0ae930 legacy=0x7f437c14dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 >> v1:172.21.15.110:6789/0 conn(0x7f437c06ba20 legacy=0x7f437c13f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c146590 con 0x7f437c06cb80 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4370001610 con 0x7f437c06cb80 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4370001930 con 0x7f437c06cb80 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4379ffb640 1 -- 172.21.15.136:0/3690954200 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4370001d00 con 0x7f437c06cb80 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.697+0000 7f4381721640 1 -- 172.21.15.136:0/3690954200 >> v1:172.21.15.136:6789/0 conn(0x7f437c06cb80 legacy=0x7f437c145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.698+0000 7f4381721640 1 -- 172.21.15.136:0/3690954200 shutdown_connections 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.698+0000 7f4381721640 1 -- 172.21.15.136:0/3690954200 >> 172.21.15.136:0/3690954200 conn(0x7f437c0b5ef0 msgr2=0x7f437c0b6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.698+0000 7f4381721640 1 -- 172.21.15.136:0/3690954200 shutdown_connections 2024-05-04T17:35:47.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.698+0000 7f4381721640 1 -- 172.21.15.136:0/3690954200 wait complete. 2024-05-04T17:35:47.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f4381721640 1 Processor -- start 2024-05-04T17:35:47.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f4381721640 1 -- start start 2024-05-04T17:35:47.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f4381721640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c135ad0 con 0x7f437c0ae930 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f4381721640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c1d52a0 con 0x7f437c06cb80 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f4381721640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c1356f0 con 0x7f437c06ba20 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f437a7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f437c06cb80 0x7f437c1d4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52772/0 (socket says 172.21.15.136:52772) 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.699+0000 7f437a7fc640 1 -- 172.21.15.136:0/2596036838 learned_addr learned my addr 172.21.15.136:0/2596036838 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1288763304 0 0) 0x7f437c1d52a0 con 0x7f437c06cb80 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4368003520 con 0x7f437c06cb80 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522903866 0 0) 0x7f437c135ad0 con 0x7f437c0ae930 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f437c1d52a0 con 0x7f437c0ae930 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2276212397 0 0) 0x7f437c1356f0 con 0x7f437c06ba20 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f437c135ad0 con 0x7f437c06ba20 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 602368797 0 0) 0x7f4368003520 con 0x7f437c06cb80 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f437c1356f0 con 0x7f437c06cb80 2024-05-04T17:35:47.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f43700015d0 con 0x7f437c06cb80 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 65178975 0 0) 0x7f437c1d52a0 con 0x7f437c0ae930 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4368003520 con 0x7f437c0ae930 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 23548907 0 0) 0x7f437c135ad0 con 0x7f437c06ba20 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f437c1d52a0 con 0x7f437c06ba20 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4374001b90 con 0x7f437c0ae930 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f436c001830 con 0x7f437c06ba20 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1459384300 0 0) 0x7f437c1356f0 con 0x7f437c06cb80 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 >> v1:172.21.15.110:6790/0 conn(0x7f437c06ba20 legacy=0x7f437c135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 >> v1:172.21.15.110:6789/0 conn(0x7f437c0ae930 legacy=0x7f437c1e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c1d4300 con 0x7f437c06cb80 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f437c1d45b0 con 0x7f437c06cb80 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f437c1e38e0 con 0x7f437c06cb80 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4370001930 con 0x7f437c06cb80 2024-05-04T17:35:47.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4370004e60 con 0x7f437c06cb80 2024-05-04T17:35:47.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4370005450 con 0x7f437c06cb80 2024-05-04T17:35:47.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.700+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 721660070 0 0) 0x7f4370006a50 con 0x7f437c06cb80 2024-05-04T17:35:47.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.701+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c005080 con 0x7f437c06cb80 2024-05-04T17:35:47.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.703+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4370005740 con 0x7f437c06cb80 2024-05-04T17:35:47.723 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:47.723+0000 7f28c840f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-04T17:35:47.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.775+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ac107878-2a9c-435d-a24b-634939607eb5", "id": 0} v 0) -- 0x7f435c005370 con 0x7f437c06cb80 2024-05-04T17:35:47.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.780+0000 7f43637fe640 1 -- 172.21.15.136:0/2596036838 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac107878-2a9c-435d-a24b-634939607eb5", "id": 0}]=0 v3) ==== 112+0+2 (unknown 3991053688 0 732707515) 0x7f43700059f0 con 0x7f437c06cb80 2024-05-04T17:35:47.782 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-05-04T17:35:47.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.781+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 >> v1:172.21.15.136:6789/0 conn(0x7f437c06cb80 legacy=0x7f437c1d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.782+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 shutdown_connections 2024-05-04T17:35:47.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.782+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 >> 172.21.15.136:0/2596036838 conn(0x7f437c0b5ef0 msgr2=0x7f437c0b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:47.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.782+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 shutdown_connections 2024-05-04T17:35:47.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.782+0000 7f4381721640 1 -- 172.21.15.136:0/2596036838 wait complete. 2024-05-04T17:35:47.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 895a609b-83a1-4d9d-a323-9e777df46326 1 2024-05-04T17:35:47.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.888+0000 7fc774100640 1 Processor -- start 2024-05-04T17:35:47.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.888+0000 7fc774100640 1 -- start start 2024-05-04T17:35:47.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc774100640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0ab350 con 0x7fc76c145f20 2024-05-04T17:35:47.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc774100640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0a98d0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc774100640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0b2410 con 0x7fc76c0b4150 2024-05-04T17:35:47.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc771e9c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc76c0b4b10 0x7fc76c13f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52784/0 (socket says 172.21.15.136:52784) 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc771e9c640 1 -- 172.21.15.136:0/2913641416 learned_addr learned my addr 172.21.15.136:0/2913641416 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1490741603 0 0) 0x7fc76c0a98d0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc758003520 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715645305 0 0) 0x7fc76c0ab350 con 0x7fc76c145f20 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c0a98d0 con 0x7fc76c145f20 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028381370 0 0) 0x7fc76c0b2410 con 0x7fc76c0b4150 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c0ab350 con 0x7fc76c0b4150 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3072811203 0 0) 0x7fc758003520 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 >> v1:172.21.15.110:6790/0 conn(0x7fc76c0b4150 legacy=0x7fc76c145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 >> v1:172.21.15.110:6789/0 conn(0x7fc76c145f20 legacy=0x7fc76c14dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c0a9df0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.889+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc75c001610 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc75c001930 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc770e9a640 1 -- 172.21.15.136:0/2913641416 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc75c001d00 con 0x7fc76c0b4b10 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc774100640 1 -- 172.21.15.136:0/2913641416 >> v1:172.21.15.136:6789/0 conn(0x7fc76c0b4b10 legacy=0x7fc76c13f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc774100640 1 -- 172.21.15.136:0/2913641416 shutdown_connections 2024-05-04T17:35:47.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc774100640 1 -- 172.21.15.136:0/2913641416 >> 172.21.15.136:0/2913641416 conn(0x7fc76c061c70 msgr2=0x7fc76c062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:47.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc774100640 1 -- 172.21.15.136:0/2913641416 shutdown_connections 2024-05-04T17:35:47.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.890+0000 7fc774100640 1 -- 172.21.15.136:0/2913641416 wait complete. 2024-05-04T17:35:47.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.891+0000 7fc774100640 1 Processor -- start 2024-05-04T17:35:47.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.891+0000 7fc774100640 1 -- start start 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc774100640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c135960 con 0x7fc76c145f20 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc774100640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c135bd0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc774100640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c135eb0 con 0x7fc76c0b4150 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc77169b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc76c0b4b10 0x7fc76c142ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52792/0 (socket says 172.21.15.136:52792) 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc77169b640 1 -- 172.21.15.136:0/1734710851 learned_addr learned my addr 172.21.15.136:0/1734710851 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1465938469 0 0) 0x7fc76c135bd0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc750003520 con 0x7fc76c0b4b10 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933075128 0 0) 0x7fc76c135960 con 0x7fc76c145f20 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c135bd0 con 0x7fc76c145f20 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883104821 0 0) 0x7fc76c135eb0 con 0x7fc76c0b4150 2024-05-04T17:35:47.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c135960 con 0x7fc76c0b4150 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1498605137 0 0) 0x7fc750003520 con 0x7fc76c0b4b10 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc76c135eb0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc768001990 con 0x7fc76c0b4b10 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3070885222 0 0) 0x7fc76c135bd0 con 0x7fc76c145f20 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc750003520 con 0x7fc76c145f20 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc764001a20 con 0x7fc76c145f20 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2775862337 0 0) 0x7fc76c135960 con 0x7fc76c0b4150 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc76c135bd0 con 0x7fc76c0b4150 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc75c001430 con 0x7fc76c0b4150 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1016312499 0 0) 0x7fc76c135eb0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 >> v1:172.21.15.110:6790/0 conn(0x7fc76c0b4150 legacy=0x7fc76c1362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.892+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 >> v1:172.21.15.110:6789/0 conn(0x7fc76c145f20 legacy=0x7fc76c1e25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c1410a0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c142590 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc76c13b3f0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7680038c0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc768003c90 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc768004240 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 3178258272 0 0) 0x7fc768006700 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.893+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc754005080 con 0x7fc76c0b4b10 2024-05-04T17:35:47.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.895+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc768001ce0 con 0x7fc76c0b4b10 2024-05-04T17:35:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.968+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "895a609b-83a1-4d9d-a323-9e777df46326", "id": 1} v 0) -- 0x7fc754005370 con 0x7fc76c0b4b10 2024-05-04T17:35:47.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.973+0000 7fc7627fc640 1 -- 172.21.15.136:0/1734710851 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "895a609b-83a1-4d9d-a323-9e777df46326", "id": 1}]=0 v4) ==== 112+0+2 (unknown 291054186 0 940483276) 0x7fc768005890 con 0x7fc76c0b4b10 2024-05-04T17:35:47.974 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-05-04T17:35:47.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.974+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 >> v1:172.21.15.136:6789/0 conn(0x7fc76c0b4b10 legacy=0x7fc76c142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:47.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.974+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 shutdown_connections 2024-05-04T17:35:47.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.974+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 >> 172.21.15.136:0/1734710851 conn(0x7fc76c061c70 msgr2=0x7fc76c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:47.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.974+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 shutdown_connections 2024-05-04T17:35:47.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:47.974+0000 7fc774100640 1 -- 172.21.15.136:0/1734710851 wait complete. 2024-05-04T17:35:47.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new be5c569a-c157-48d0-b8d0-f44918cb1204 2 2024-05-04T17:35:48.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.083+0000 7fe4b69c7640 1 Processor -- start 2024-05-04T17:35:48.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.083+0000 7fe4b69c7640 1 -- start start 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4b69c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b00733a0 con 0x7fe4b0140010 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4b69c7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b0073680 con 0x7fe4b0140480 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4b69c7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b013db50 con 0x7fe4b0074b60 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4b4f64640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe4b0140480 0x7fe4b014c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52808/0 (socket says 172.21.15.136:52808) 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4b4f64640 1 -- 172.21.15.136:0/323528095 learned_addr learned my addr 172.21.15.136:0/323528095 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279985723 0 0) 0x7fe4b0073680 con 0x7fe4b0140480 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe498003520 con 0x7fe4b0140480 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349200976 0 0) 0x7fe4b013db50 con 0x7fe4b0074b60 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4b0073680 con 0x7fe4b0074b60 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2654817246 0 0) 0x7fe4b00733a0 con 0x7fe4b0140010 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4b013db50 con 0x7fe4b0140010 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1558692745 0 0) 0x7fe498003520 con 0x7fe4b0140480 2024-05-04T17:35:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 >> v1:172.21.15.110:6790/0 conn(0x7fe4b0074b60 legacy=0x7fe4b013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 >> v1:172.21.15.110:6789/0 conn(0x7fe4b0140010 legacy=0x7fe4b0148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b0072010 con 0x7fe4b0140480 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe4a8001990 con 0x7fe4b0140480 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4a8001cb0 con 0x7fe4b0140480 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.084+0000 7fe4aeffd640 1 -- 172.21.15.136:0/323528095 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe4a80020c0 con 0x7fe4b0140480 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.085+0000 7fe4b69c7640 1 -- 172.21.15.136:0/323528095 >> v1:172.21.15.136:6789/0 conn(0x7fe4b0140480 legacy=0x7fe4b014c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.085+0000 7fe4b69c7640 1 -- 172.21.15.136:0/323528095 shutdown_connections 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.085+0000 7fe4b69c7640 1 -- 172.21.15.136:0/323528095 >> 172.21.15.136:0/323528095 conn(0x7fe4b00b0bb0 msgr2=0x7fe4b00b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.085+0000 7fe4b69c7640 1 -- 172.21.15.136:0/323528095 shutdown_connections 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.085+0000 7fe4b69c7640 1 -- 172.21.15.136:0/323528095 wait complete. 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.086+0000 7fe4b69c7640 1 Processor -- start 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.086+0000 7fe4b69c7640 1 -- start start 2024-05-04T17:35:48.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4b69c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b01d50d0 con 0x7fe4b0074b60 2024-05-04T17:35:48.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4b69c7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b01d5d00 con 0x7fe4b0140010 2024-05-04T17:35:48.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4b69c7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b011cb00 con 0x7fe4b0140480 2024-05-04T17:35:48.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4af7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe4b0140010 0x7fe4b011f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52818/0 (socket says 172.21.15.136:52818) 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4af7fe640 1 -- 172.21.15.136:0/3615489971 learned_addr learned my addr 172.21.15.136:0/3615489971 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242837362 0 0) 0x7fe4b01d5d00 con 0x7fe4b0140010 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49c003520 con 0x7fe4b0140010 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2834918528 0 0) 0x7fe4b01d50d0 con 0x7fe4b0074b60 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4b01d5d00 con 0x7fe4b0074b60 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3807154174 0 0) 0x7fe4b011cb00 con 0x7fe4b0140480 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4b01d50d0 con 0x7fe4b0140480 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 249479710 0 0) 0x7fe49c003520 con 0x7fe4b0140010 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4b011cb00 con 0x7fe4b0140010 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe4a40019d0 con 0x7fe4b0140010 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2996506691 0 0) 0x7fe4b01d5d00 con 0x7fe4b0074b60 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe49c003520 con 0x7fe4b0074b60 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2944878395 0 0) 0x7fe4b01d50d0 con 0x7fe4b0140480 2024-05-04T17:35:48.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4b01d5d00 con 0x7fe4b0140480 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe4a00019f0 con 0x7fe4b0074b60 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe4a80017b0 con 0x7fe4b0140480 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 162099438 0 0) 0x7fe4b011cb00 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 >> v1:172.21.15.110:6790/0 conn(0x7fe4b0140480 legacy=0x7fe4b01d55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 >> v1:172.21.15.110:6789/0 conn(0x7fe4b0074b60 legacy=0x7fe4b01202f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b011ccd0 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b013b670 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.087+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4b013bb80 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.088+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4a40038a0 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.088+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe4a4003bf0 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.088+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe4a40041e0 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.088+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 2594975961 0 0) 0x7fe4a4006a00 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.088+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe490005080 con 0x7fe4b0140010 2024-05-04T17:35:48.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.090+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe4a40065e0 con 0x7fe4b0140010 2024-05-04T17:35:48.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.163+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "be5c569a-c157-48d0-b8d0-f44918cb1204", "id": 2} v 0) -- 0x7fe490005370 con 0x7fe4b0140010 2024-05-04T17:35:48.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.171+0000 7fe4acff9640 1 -- 172.21.15.136:0/3615489971 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be5c569a-c157-48d0-b8d0-f44918cb1204", "id": 2}]=0 v5) ==== 112+0+2 (unknown 2385952296 0 216599125) 0x7fe4a4005780 con 0x7fe4b0140010 2024-05-04T17:35:48.172 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-05-04T17:35:48.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.172+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 >> v1:172.21.15.136:6789/0 conn(0x7fe4b0140010 legacy=0x7fe4b011f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.172+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 shutdown_connections 2024-05-04T17:35:48.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.172+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 >> 172.21.15.136:0/3615489971 conn(0x7fe4b00b0bb0 msgr2=0x7fe4b013e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.172+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 shutdown_connections 2024-05-04T17:35:48.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.172+0000 7fe4b69c7640 1 -- 172.21.15.136:0/3615489971 wait complete. 2024-05-04T17:35:48.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ff7aaeae-b017-4ece-8dab-2adea4056df6 3 2024-05-04T17:35:48.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.280+0000 7f1bbe2cf640 1 Processor -- start 2024-05-04T17:35:48.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.280+0000 7f1bbe2cf640 1 -- start start 2024-05-04T17:35:48.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.280+0000 7f1bbe2cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb8073260 con 0x7f1bb813d980 2024-05-04T17:35:48.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.280+0000 7f1bbe2cf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb80b0f00 con 0x7f1bb8143cc0 2024-05-04T17:35:48.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.280+0000 7f1bbe2cf640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb80b11e0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bbc86c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1bb8143cc0 0x7f1bb814b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52834/0 (socket says 172.21.15.136:52834) 2024-05-04T17:35:48.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bbc86c640 1 -- 172.21.15.136:0/2243512569 learned_addr learned my addr 172.21.15.136:0/2243512569 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058998908 0 0) 0x7f1bb80b0f00 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ba0003520 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208014084 0 0) 0x7f1bb80b11e0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bb80b0f00 con 0x7f1bb8061ee0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331573364 0 0) 0x7f1bb8073260 con 0x7f1bb813d980 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bb80b11e0 con 0x7f1bb813d980 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2499556747 0 0) 0x7f1ba0003520 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 >> v1:172.21.15.110:6790/0 conn(0x7f1bb8061ee0 legacy=0x7f1bb813d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 >> v1:172.21.15.110:6789/0 conn(0x7f1bb813d980 legacy=0x7f1bb81435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb8072d40 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1bb0001990 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bb0001c40 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.281+0000 7f1bb6ffd640 1 -- 172.21.15.136:0/2243512569 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1bb0002010 con 0x7f1bb8143cc0 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.282+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/2243512569 >> v1:172.21.15.136:6789/0 conn(0x7f1bb8143cc0 legacy=0x7f1bb814b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.282+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/2243512569 shutdown_connections 2024-05-04T17:35:48.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.282+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/2243512569 >> 172.21.15.136:0/2243512569 conn(0x7f1bb80b4af0 msgr2=0x7f1bb80b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.282+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/2243512569 shutdown_connections 2024-05-04T17:35:48.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.282+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/2243512569 wait complete. 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.283+0000 7f1bbe2cf640 1 Processor -- start 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.283+0000 7f1bbe2cf640 1 -- start start 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.283+0000 7f1bbe2cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb813fc50 con 0x7f1bb8143cc0 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.283+0000 7f1bbe2cf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb8136c20 con 0x7f1bb8061ee0 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.283+0000 7f1bbe2cf640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb8136f00 con 0x7f1bb813d980 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1bb8061ee0 0x7f1bb8135420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52840/0 (socket says 172.21.15.136:52840) 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb7fff640 1 -- 172.21.15.136:0/4229205351 learned_addr learned my addr 172.21.15.136:0/4229205351 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816208458 0 0) 0x7f1bb8136c20 con 0x7f1bb8061ee0 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ba4003520 con 0x7f1bb8061ee0 2024-05-04T17:35:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2183177044 0 0) 0x7f1bb813fc50 con 0x7f1bb8143cc0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bb8136c20 con 0x7f1bb8143cc0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713645467 0 0) 0x7f1bb8136f00 con 0x7f1bb813d980 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bb813fc50 con 0x7f1bb813d980 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 505228398 0 0) 0x7f1ba4003520 con 0x7f1bb8061ee0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bb8136f00 con 0x7f1bb8061ee0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ba80018b0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 171712145 0 0) 0x7f1bb8136c20 con 0x7f1bb8143cc0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ba4003520 con 0x7f1bb8143cc0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1bb0001df0 con 0x7f1bb8143cc0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3174243909 0 0) 0x7f1bb813fc50 con 0x7f1bb813d980 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bb8136c20 con 0x7f1bb813d980 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1bac0018b0 con 0x7f1bb813d980 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 853726195 0 0) 0x7f1bb8136f00 con 0x7f1bb8061ee0 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 >> v1:172.21.15.110:6790/0 conn(0x7f1bb813d980 legacy=0x7f1bb813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 >> v1:172.21.15.110:6789/0 conn(0x7f1bb8143cc0 legacy=0x7f1bb81e00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb8135c20 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb8134830 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1bb8138710 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ba8001ce0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.284+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ba8003a60 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.285+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1ba8003fd0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.285+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 3015485856 0 0) 0x7f1ba80042c0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.285+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b98005080 con 0x7f1bb8061ee0 2024-05-04T17:35:48.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.287+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1ba80057c0 con 0x7f1bb8061ee0 2024-05-04T17:35:48.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.360+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ff7aaeae-b017-4ece-8dab-2adea4056df6", "id": 3} v 0) -- 0x7f1b98005370 con 0x7f1bb8061ee0 2024-05-04T17:35:48.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.367+0000 7f1bb4ff9640 1 -- 172.21.15.136:0/4229205351 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff7aaeae-b017-4ece-8dab-2adea4056df6", "id": 3}]=0 v6) ==== 112+0+2 (unknown 1198788469 0 525046306) 0x7f1ba8005a20 con 0x7f1bb8061ee0 2024-05-04T17:35:48.368 INFO:teuthology.orchestra.run.smithi136.stdout:3 2024-05-04T17:35:48.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.368+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 >> v1:172.21.15.136:6789/0 conn(0x7f1bb8061ee0 legacy=0x7f1bb8135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.368+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 shutdown_connections 2024-05-04T17:35:48.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.368+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 >> 172.21.15.136:0/4229205351 conn(0x7f1bb80b4af0 msgr2=0x7f1bb8142b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.368+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 shutdown_connections 2024-05-04T17:35:48.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.368+0000 7f1bbe2cf640 1 -- 172.21.15.136:0/4229205351 wait complete. 2024-05-04T17:35:48.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 98a44aae-f478-4906-9f02-c63ae35f54d4 4 2024-05-04T17:35:48.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.382+0000 7f3cc3014040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-04T17:35:48.441 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.441+0000 7f28c840f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-04T17:35:48.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.467+0000 7f3cc3014040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-04T17:35:48.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.476+0000 7f3d7ecf2640 1 Processor -- start 2024-05-04T17:35:48.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.476+0000 7f3d7ecf2640 1 -- start start 2024-05-04T17:35:48.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.476+0000 7f3d7ecf2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d780ab350 con 0x7f3d780b4b10 2024-05-04T17:35:48.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.476+0000 7f3d7ecf2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d780a98d0 con 0x7f3d780b4150 2024-05-04T17:35:48.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.476+0000 7f3d7ecf2640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d780b2410 con 0x7f3d78145f20 2024-05-04T17:35:48.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d77fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3d780b4150 0x7f3d78145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52852/0 (socket says 172.21.15.136:52852) 2024-05-04T17:35:48.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d77fff640 1 -- 172.21.15.136:0/162601894 learned_addr learned my addr 172.21.15.136:0/162601894 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872492262 0 0) 0x7f3d780a98d0 con 0x7f3d780b4150 2024-05-04T17:35:48.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d68003520 con 0x7f3d780b4150 2024-05-04T17:35:48.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1830406914 0 0) 0x7f3d780ab350 con 0x7f3d780b4b10 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d780a98d0 con 0x7f3d780b4b10 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089986555 0 0) 0x7f3d780b2410 con 0x7f3d78145f20 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d780ab350 con 0x7f3d78145f20 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 195599205 0 0) 0x7f3d68003520 con 0x7f3d780b4150 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 >> v1:172.21.15.110:6790/0 conn(0x7f3d78145f20 legacy=0x7f3d7814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 >> v1:172.21.15.110:6789/0 conn(0x7f3d780b4b10 legacy=0x7f3d7813f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d780a9df0 con 0x7f3d780b4150 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3d64001610 con 0x7f3d780b4150 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d64001930 con 0x7f3d780b4150 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.477+0000 7f3d777fe640 1 -- 172.21.15.136:0/162601894 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3d64001d00 con 0x7f3d780b4150 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.478+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/162601894 >> v1:172.21.15.136:6789/0 conn(0x7f3d780b4150 legacy=0x7f3d78145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.478+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/162601894 shutdown_connections 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.478+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/162601894 >> 172.21.15.136:0/162601894 conn(0x7f3d78061c70 msgr2=0x7f3d78062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.478+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/162601894 shutdown_connections 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.478+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/162601894 wait complete. 2024-05-04T17:35:48.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ecf2640 1 Processor -- start 2024-05-04T17:35:48.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ecf2640 1 -- start start 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ecf2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d78134c10 con 0x7f3d78145f20 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ecf2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d78133d60 con 0x7f3d780b4150 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ecf2640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d78142880 con 0x7f3d780b4b10 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ca8e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3d780b4150 0x7f3d78140090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52862/0 (socket says 172.21.15.136:52862) 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.479+0000 7f3d7ca8e640 1 -- 172.21.15.136:0/1990541528 learned_addr learned my addr 172.21.15.136:0/1990541528 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d7d28f640 1 -- 172.21.15.136:0/1990541528 >> v1:172.21.15.110:6789/0 conn(0x7f3d78145f20 legacy=0x7f3d781e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d7d28f640 1 -- 172.21.15.136:0/1990541528 >> v1:172.21.15.110:6789/0 conn(0x7f3d78145f20 legacy=0x7f3d781e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d7d28f640 1 --1- 172.21.15.136:0/1990541528 >> v1:172.21.15.110:6789/0 conn(0x7f3d78145f20 0x7f3d781e3010 :-1 s=OPENED pgs=29 cs=1 l=1).handle_message read tag failed 2024-05-04T17:35:48.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d7d28f640 1 --1- 172.21.15.136:0/1990541528 >> v1:172.21.15.110:6789/0 conn(0x7f3d78145f20 0x7f3d781e3010 :-1 s=OPENED pgs=29 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970936057 0 0) 0x7f3d78133d60 con 0x7f3d780b4150 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d5c003520 con 0x7f3d780b4150 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221574823 0 0) 0x7f3d78142880 con 0x7f3d780b4b10 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d78133d60 con 0x7f3d780b4b10 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3162272265 0 0) 0x7f3d5c003520 con 0x7f3d780b4150 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d78142880 con 0x7f3d780b4150 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3d6c0018b0 con 0x7f3d780b4150 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2479148513 0 0) 0x7f3d78133d60 con 0x7f3d780b4b10 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d5c003520 con 0x7f3d780b4b10 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3d64001430 con 0x7f3d780b4b10 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1709994126 0 0) 0x7f3d78142880 con 0x7f3d780b4150 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 >> v1:172.21.15.110:6790/0 conn(0x7f3d780b4b10 legacy=0x7f3d78133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 >> v1:172.21.15.110:6789/0 conn(0x7f3d78145f20 legacy=0x7f3d781e3010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:35:48.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d781407a0 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d781424a0 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d78138b30 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d6c001ce0 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3d6c003a60 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3d6c003fd0 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.480+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 4147276492 0 0) 0x7f3d6c006940 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.481+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d58005080 con 0x7f3d780b4150 2024-05-04T17:35:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.483+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3d6c004450 con 0x7f3d780b4150 2024-05-04T17:35:48.511 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.509+0000 7f3cc3014040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-04T17:35:48.534 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.533+0000 7f28c840f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-04T17:35:48.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.555+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "98a44aae-f478-4906-9f02-c63ae35f54d4", "id": 4} v 0) -- 0x7f3d58005370 con 0x7f3d780b4150 2024-05-04T17:35:48.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.563+0000 7f3d757fa640 1 -- 172.21.15.136:0/1990541528 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98a44aae-f478-4906-9f02-c63ae35f54d4", "id": 4}]=0 v7) ==== 112+0+2 (unknown 66311648 0 1697012583) 0x7f3d6c006500 con 0x7f3d780b4150 2024-05-04T17:35:48.564 INFO:teuthology.orchestra.run.smithi136.stdout:4 2024-05-04T17:35:48.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.564+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 >> v1:172.21.15.136:6789/0 conn(0x7f3d780b4150 legacy=0x7f3d78140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.564+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 shutdown_connections 2024-05-04T17:35:48.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.564+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 >> 172.21.15.136:0/1990541528 conn(0x7f3d78061c70 msgr2=0x7f3d78138fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.564+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 shutdown_connections 2024-05-04T17:35:48.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.564+0000 7f3d7ecf2640 1 -- 172.21.15.136:0/1990541528 wait complete. 2024-05-04T17:35:48.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new c625ba28-d2ca-4ecb-8de2-4fecc934e626 5 2024-05-04T17:35:48.579 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.579+0000 7f28c840f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-04T17:35:48.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.672+0000 7f6c2c231640 1 Processor -- start 2024-05-04T17:35:48.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.672+0000 7f6c2c231640 1 -- start start 2024-05-04T17:35:48.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.672+0000 7f6c2c231640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c240ab350 con 0x7f6c24145f20 2024-05-04T17:35:48.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.672+0000 7f6c2c231640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c240a98d0 con 0x7f6c240b4b10 2024-05-04T17:35:48.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.672+0000 7f6c2c231640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c240b2410 con 0x7f6c240b4150 2024-05-04T17:35:48.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c29fcd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6c240b4b10 0x7f6c2413f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52864/0 (socket says 172.21.15.136:52864) 2024-05-04T17:35:48.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c29fcd640 1 -- 172.21.15.136:0/3867979360 learned_addr learned my addr 172.21.15.136:0/3867979360 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167455309 0 0) 0x7f6c240a98d0 con 0x7f6c240b4b10 2024-05-04T17:35:48.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c10003520 con 0x7f6c240b4b10 2024-05-04T17:35:48.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 214020914 0 0) 0x7f6c240b2410 con 0x7f6c240b4150 2024-05-04T17:35:48.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c240a98d0 con 0x7f6c240b4150 2024-05-04T17:35:48.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1932450914 0 0) 0x7f6c10003520 con 0x7f6c240b4b10 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 >> v1:172.21.15.110:6790/0 conn(0x7f6c240b4150 legacy=0x7f6c24145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 >> v1:172.21.15.110:6789/0 conn(0x7f6c24145f20 legacy=0x7f6c2414dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c240a9df0 con 0x7f6c240b4b10 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6c20001610 con 0x7f6c240b4b10 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c20001870 con 0x7f6c240b4b10 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.673+0000 7f6c28fcb640 1 -- 172.21.15.136:0/3867979360 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6c20001c80 con 0x7f6c240b4b10 2024-05-04T17:35:48.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.674+0000 7f6c2c231640 1 -- 172.21.15.136:0/3867979360 >> v1:172.21.15.136:6789/0 conn(0x7f6c240b4b10 legacy=0x7f6c2413f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.674+0000 7f6c2c231640 1 -- 172.21.15.136:0/3867979360 shutdown_connections 2024-05-04T17:35:48.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.674+0000 7f6c2c231640 1 -- 172.21.15.136:0/3867979360 >> 172.21.15.136:0/3867979360 conn(0x7f6c24061c70 msgr2=0x7f6c24062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.674+0000 7f6c2c231640 1 -- 172.21.15.136:0/3867979360 shutdown_connections 2024-05-04T17:35:48.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.674+0000 7f6c2c231640 1 -- 172.21.15.136:0/3867979360 wait complete. 2024-05-04T17:35:48.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.673+0000 7f3cc3014040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c2c231640 1 Processor -- start 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c2c231640 1 -- start start 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c2c231640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2413b840 con 0x7f6c240b4150 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c2c231640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2413c930 con 0x7f6c240b4b10 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c2c231640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c24142750 con 0x7f6c24145f20 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c297cc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6c240b4b10 0x7f6c241d7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52876/0 (socket says 172.21.15.136:52876) 2024-05-04T17:35:48.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.675+0000 7f6c297cc640 1 -- 172.21.15.136:0/1098858453 learned_addr learned my addr 172.21.15.136:0/1098858453 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2863444876 0 0) 0x7f6c2413c930 con 0x7f6c240b4b10 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c08003520 con 0x7f6c240b4b10 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26999318 0 0) 0x7f6c24142750 con 0x7f6c24145f20 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c2413c930 con 0x7f6c24145f20 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 537645292 0 0) 0x7f6c2413b840 con 0x7f6c240b4150 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c24142750 con 0x7f6c240b4150 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2513243019 0 0) 0x7f6c08003520 con 0x7f6c240b4b10 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c2413b840 con 0x7f6c240b4b10 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6c14001900 con 0x7f6c240b4b10 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 566369960 0 0) 0x7f6c2413c930 con 0x7f6c24145f20 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c08003520 con 0x7f6c24145f20 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6c1c001b10 con 0x7f6c24145f20 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 162634827 0 0) 0x7f6c2413b840 con 0x7f6c240b4b10 2024-05-04T17:35:48.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 >> v1:172.21.15.110:6790/0 conn(0x7f6c24145f20 legacy=0x7f6c241d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 >> v1:172.21.15.110:6789/0 conn(0x7f6c240b4150 legacy=0x7f6c2413c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c24142920 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c241e32e0 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c241e3690 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c14001cd0 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6c14003a60 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6c14004010 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.676+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 159881324 0 0) 0x7f6c140068a0 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.677+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c005080 con 0x7f6c240b4b10 2024-05-04T17:35:48.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.679+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6c14006bb0 con 0x7f6c240b4b10 2024-05-04T17:35:48.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.721+0000 7f3cc3014040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-04T17:35:48.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.752+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c625ba28-d2ca-4ecb-8de2-4fecc934e626", "id": 5} v 0) -- 0x7f6c0c005370 con 0x7f6c240b4b10 2024-05-04T17:35:48.757 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.756+0000 7f28c840f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-04T17:35:48.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.759+0000 7f6c1a7fc640 1 -- 172.21.15.136:0/1098858453 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c625ba28-d2ca-4ecb-8de2-4fecc934e626", "id": 5}]=0 v8) ==== 112+0+2 (unknown 3209349311 0 1988412176) 0x7f6c14006320 con 0x7f6c240b4b10 2024-05-04T17:35:48.760 INFO:teuthology.orchestra.run.smithi136.stdout:5 2024-05-04T17:35:48.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.760+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 >> v1:172.21.15.136:6789/0 conn(0x7f6c240b4b10 legacy=0x7f6c241d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.760+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 shutdown_connections 2024-05-04T17:35:48.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.760+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 >> 172.21.15.136:0/1098858453 conn(0x7f6c24061c70 msgr2=0x7f6c240b7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.760+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 shutdown_connections 2024-05-04T17:35:48.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.760+0000 7f6c2c231640 1 -- 172.21.15.136:0/1098858453 wait complete. 2024-05-04T17:35:48.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373 6 2024-05-04T17:35:48.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.774+0000 7f3cc3014040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-04T17:35:48.812 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.811+0000 7f28c840f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-04T17:35:48.868 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.867+0000 7f28c840f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-04T17:35:48.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:48.883+0000 7f3cc3014040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:35:48.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.908+0000 7f400754d640 1 Processor -- start 2024-05-04T17:35:48.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.908+0000 7f400754d640 1 -- start start 2024-05-04T17:35:48.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f400754d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40000ab350 con 0x7f40000b4150 2024-05-04T17:35:48.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f400754d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40000a98d0 con 0x7f4000145f20 2024-05-04T17:35:48.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f400754d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40000b2410 con 0x7f40000b4b10 2024-05-04T17:35:48.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f4005aea640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4000145f20 0x7f400014dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52880/0 (socket says 172.21.15.136:52880) 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f4005aea640 1 -- 172.21.15.136:0/223538870 learned_addr learned my addr 172.21.15.136:0/223538870 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443104323 0 0) 0x7f40000a98d0 con 0x7f4000145f20 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fec003520 con 0x7f4000145f20 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f40052e9640 1 -- 172.21.15.136:0/223538870 >> v1:172.21.15.110:6790/0 conn(0x7f40000b4b10 legacy=0x7f400013f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f40052e9640 1 -- 172.21.15.136:0/223538870 >> v1:172.21.15.110:6790/0 conn(0x7f40000b4b10 legacy=0x7f400013f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f40052e9640 1 --1- 172.21.15.136:0/223538870 >> v1:172.21.15.110:6790/0 conn(0x7f40000b4b10 0x7f400013f610 :-1 s=OPENED pgs=31 cs=1 l=1).handle_message read tag failed 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f40052e9640 1 --1- 172.21.15.136:0/223538870 >> v1:172.21.15.110:6790/0 conn(0x7f40000b4b10 0x7f400013f610 :-1 s=OPENED pgs=31 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993187650 0 0) 0x7f40000ab350 con 0x7f40000b4150 2024-05-04T17:35:48.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40000a98d0 con 0x7f40000b4150 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 598932055 0 0) 0x7f3fec003520 con 0x7f4000145f20 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 >> v1:172.21.15.110:6790/0 conn(0x7f40000b4b10 legacy=0x7f400013f610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 >> v1:172.21.15.110:6789/0 conn(0x7f40000b4150 legacy=0x7f4000145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40000a9df0 con 0x7f4000145f20 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.909+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3ff8001990 con 0x7f4000145f20 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ff8001cb0 con 0x7f4000145f20 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f3ff7fff640 1 -- 172.21.15.136:0/223538870 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3ff8002080 con 0x7f4000145f20 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f400754d640 1 -- 172.21.15.136:0/223538870 >> v1:172.21.15.136:6789/0 conn(0x7f4000145f20 legacy=0x7f400014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f400754d640 1 -- 172.21.15.136:0/223538870 shutdown_connections 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f400754d640 1 -- 172.21.15.136:0/223538870 >> 172.21.15.136:0/223538870 conn(0x7f4000061c70 msgr2=0x7f4000062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f400754d640 1 -- 172.21.15.136:0/223538870 shutdown_connections 2024-05-04T17:35:48.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.910+0000 7f400754d640 1 -- 172.21.15.136:0/223538870 wait complete. 2024-05-04T17:35:48.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.911+0000 7f400754d640 1 Processor -- start 2024-05-04T17:35:48.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.911+0000 7f400754d640 1 -- start start 2024-05-04T17:35:48.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f400754d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40001d52f0 con 0x7f4000145f20 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f400754d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40001d7880 con 0x7f40000b4b10 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f400754d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4000134e00 con 0x7f40000b4150 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f4004ae8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f40000b4b10 0x7f40001353c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52890/0 (socket says 172.21.15.136:52890) 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f4004ae8640 1 -- 172.21.15.136:0/191159864 learned_addr learned my addr 172.21.15.136:0/191159864 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3789698129 0 0) 0x7f40001d7880 con 0x7f40000b4b10 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fe4003520 con 0x7f40000b4b10 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2519342626 0 0) 0x7f4000134e00 con 0x7f40000b4150 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40001d7880 con 0x7f40000b4150 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980310668 0 0) 0x7f40001d52f0 con 0x7f4000145f20 2024-05-04T17:35:48.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4000134e00 con 0x7f4000145f20 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3762756237 0 0) 0x7f40001d7880 con 0x7f40000b4150 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40001d52f0 con 0x7f40000b4150 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3710639977 0 0) 0x7f3fe4003520 con 0x7f40000b4b10 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40001d7880 con 0x7f40000b4b10 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1170563495 0 0) 0x7f4000134e00 con 0x7f4000145f20 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fe4003520 con 0x7f4000145f20 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3ff00018a0 con 0x7f40000b4150 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3ffc001840 con 0x7f40000b4b10 2024-05-04T17:35:48.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.912+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3ff80017b0 con 0x7f4000145f20 2024-05-04T17:35:48.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 88194012 0 0) 0x7f40001d52f0 con 0x7f40000b4150 2024-05-04T17:35:48.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 >> v1:172.21.15.136:6789/0 conn(0x7f40000b4b10 legacy=0x7f40001353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 >> v1:172.21.15.110:6789/0 conn(0x7f4000145f20 legacy=0x7f40001e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40001d84b0 con 0x7f40000b4150 2024-05-04T17:35:48.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4000135ad0 con 0x7f40000b4150 2024-05-04T17:35:48.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ff0001cb0 con 0x7f40000b4150 2024-05-04T17:35:48.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3ff0003a40 con 0x7f40000b4150 2024-05-04T17:35:48.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3ff0003ff0 con 0x7f40000b4150 2024-05-04T17:35:48.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f40001e3650 con 0x7f40000b4150 2024-05-04T17:35:48.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.913+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 2601226041 0 0) 0x7f3ff00042e0 con 0x7f40000b4150 2024-05-04T17:35:48.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.914+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4000136c80 con 0x7f40000b4150 2024-05-04T17:35:48.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.916+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3ff00057c0 con 0x7f40000b4150 2024-05-04T17:35:48.984 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:48.983+0000 7f28c840f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:35:48.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.987+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd new", "uuid": "ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373", "id": 6} v 0) -- 0x7f400004fe40 con 0x7f40000b4150 2024-05-04T17:35:48.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.989+0000 7f3ff5ffb640 1 -- 172.21.15.136:0/191159864 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373", "id": 6}]=0 v9) ==== 112+0+2 (unknown 1055616161 0 1113811849) 0x7f3ff0005a20 con 0x7f40000b4150 2024-05-04T17:35:48.991 INFO:teuthology.orchestra.run.smithi136.stdout:6 2024-05-04T17:35:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.990+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 >> v1:172.21.15.110:6790/0 conn(0x7f40000b4150 legacy=0x7f40001d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.990+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 shutdown_connections 2024-05-04T17:35:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.990+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 >> 172.21.15.136:0/191159864 conn(0x7f4000061c70 msgr2=0x7f40000b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.991+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 shutdown_connections 2024-05-04T17:35:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:48.991+0000 7f400754d640 1 -- 172.21.15.136:0/191159864 wait complete. 2024-05-04T17:35:49.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new db593666-b9e5-4cee-b022-f243b58345cc 7 2024-05-04T17:35:49.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.098+0000 7f9b6aa9a640 1 Processor -- start 2024-05-04T17:35:49.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.098+0000 7f9b6aa9a640 1 -- start start 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b6aa9a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b64072e30 con 0x7f9b64144240 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b6aa9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b640b2080 con 0x7f9b6413da10 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b6aa9a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b64143d20 con 0x7f9b640b1880 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b69037640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b6413da10 0x7f9b6414b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52894/0 (socket says 172.21.15.136:52894) 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b69037640 1 -- 172.21.15.136:0/2910757456 learned_addr learned my addr 172.21.15.136:0/2910757456 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b63fff640 1 -- 172.21.15.136:0/2910757456 >> v1:172.21.15.110:6789/0 conn(0x7f9b64144240 legacy=0x7f9b6413d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b63fff640 1 -- 172.21.15.136:0/2910757456 >> v1:172.21.15.110:6789/0 conn(0x7f9b64144240 legacy=0x7f9b6413d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b63fff640 1 --1- 172.21.15.136:0/2910757456 >> v1:172.21.15.110:6789/0 conn(0x7f9b64144240 0x7f9b6413d300 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086387665 0 0) 0x7f9b640b2080 con 0x7f9b6413da10 2024-05-04T17:35:49.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b54003520 con 0x7f9b6413da10 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2381943358 0 0) 0x7f9b64143d20 con 0x7f9b640b1880 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.099+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b640b2080 con 0x7f9b640b1880 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2141496120 0 0) 0x7f9b54003520 con 0x7f9b6413da10 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 >> v1:172.21.15.110:6790/0 conn(0x7f9b640b1880 legacy=0x7f9b64070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 >> v1:172.21.15.110:6789/0 conn(0x7f9b64144240 legacy=0x7f9b6413d300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b64070b90 con 0x7f9b6413da10 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f9b5c001990 con 0x7f9b6413da10 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b5c001c40 con 0x7f9b6413da10 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b637fe640 1 -- 172.21.15.136:0/2910757456 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f9b5c002050 con 0x7f9b6413da10 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/2910757456 >> v1:172.21.15.136:6789/0 conn(0x7f9b6413da10 legacy=0x7f9b6414b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/2910757456 shutdown_connections 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/2910757456 >> 172.21.15.136:0/2910757456 conn(0x7f9b640b2700 msgr2=0x7f9b640b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:49.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/2910757456 shutdown_connections 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.100+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/2910757456 wait complete. 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.101+0000 7f9b6aa9a640 1 Processor -- start 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.101+0000 7f9b6aa9a640 1 -- start start 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b6aa9a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b641400a0 con 0x7f9b64144240 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b6aa9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b64140dc0 con 0x7f9b6413da10 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b6aa9a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6413fce0 con 0x7f9b640b1880 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b63fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b6413da10 0x7f9b641d5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52904/0 (socket says 172.21.15.136:52904) 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b63fff640 1 -- 172.21.15.136:0/82453537 learned_addr learned my addr 172.21.15.136:0/82453537 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265458515 0 0) 0x7f9b64140dc0 con 0x7f9b6413da10 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b48003520 con 0x7f9b6413da10 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145309870 0 0) 0x7f9b6413fce0 con 0x7f9b640b1880 2024-05-04T17:35:49.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.102+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b64140dc0 con 0x7f9b640b1880 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196801369 0 0) 0x7f9b641400a0 con 0x7f9b64144240 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b6413fce0 con 0x7f9b64144240 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2603483156 0 0) 0x7f9b48003520 con 0x7f9b6413da10 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b641400a0 con 0x7f9b6413da10 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f9b50004020 con 0x7f9b6413da10 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3619688768 0 0) 0x7f9b64140dc0 con 0x7f9b640b1880 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b48003520 con 0x7f9b640b1880 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f9b58001890 con 0x7f9b640b1880 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1889947907 0 0) 0x7f9b6413fce0 con 0x7f9b64144240 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b64140dc0 con 0x7f9b64144240 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f9b5c001df0 con 0x7f9b64144240 2024-05-04T17:35:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1900913642 0 0) 0x7f9b641400a0 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 >> v1:172.21.15.110:6790/0 conn(0x7f9b640b1880 legacy=0x7f9b64134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 >> v1:172.21.15.110:6789/0 conn(0x7f9b64144240 legacy=0x7f9b641e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6413feb0 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b64140380 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b641428f0 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b50004410 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f9b50001320 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b500018d0 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.103+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (unknown 4209044807 0 0) 0x7f9b50001320 con 0x7f9b6413da10 2024-05-04T17:35:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.104+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b44005080 con 0x7f9b6413da10 2024-05-04T17:35:49.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.106+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b50006b60 con 0x7f9b6413da10 2024-05-04T17:35:49.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.178+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "db593666-b9e5-4cee-b022-f243b58345cc", "id": 7} v 0) -- 0x7f9b44005370 con 0x7f9b6413da10 2024-05-04T17:35:49.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.181+0000 7f9b617fa640 1 -- 172.21.15.136:0/82453537 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "db593666-b9e5-4cee-b022-f243b58345cc", "id": 7}]=0 v10) ==== 112+0+2 (unknown 1847861028 0 1371665406) 0x7f9b500063e0 con 0x7f9b6413da10 2024-05-04T17:35:49.182 INFO:teuthology.orchestra.run.smithi136.stdout:7 2024-05-04T17:35:49.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.182+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 >> v1:172.21.15.136:6789/0 conn(0x7f9b6413da10 legacy=0x7f9b641d5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.182+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 shutdown_connections 2024-05-04T17:35:49.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.182+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 >> 172.21.15.136:0/82453537 conn(0x7f9b640b2700 msgr2=0x7f9b6413d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:49.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.182+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 shutdown_connections 2024-05-04T17:35:49.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-04T17:35:49.182+0000 7f9b6aa9a640 1 -- 172.21.15.136:0/82453537 wait complete. 2024-05-04T17:35:49.193 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-04T17:35:49.194 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-04T17:35:49.196 INFO:tasks.ceph.osd.0:Started 2024-05-04T17:35:49.196 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-04T17:35:49.196 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-04T17:35:49.198 INFO:tasks.ceph.osd.1:Started 2024-05-04T17:35:49.198 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-04T17:35:49.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-04T17:35:49.200 INFO:tasks.ceph.osd.2:Started 2024-05-04T17:35:49.201 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-04T17:35:49.201 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-04T17:35:49.206 INFO:tasks.ceph.osd.3:Started 2024-05-04T17:35:49.206 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-04T17:35:49.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-04T17:35:49.208 INFO:tasks.ceph.osd.4:Started 2024-05-04T17:35:49.208 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-04T17:35:49.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-04T17:35:49.211 INFO:tasks.ceph.osd.5:Started 2024-05-04T17:35:49.211 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-04T17:35:49.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-04T17:35:49.213 INFO:tasks.ceph.osd.6:Started 2024-05-04T17:35:49.213 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-04T17:35:49.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-04T17:35:49.215 INFO:tasks.ceph.osd.7:Started 2024-05-04T17:35:49.216 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:35:49.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:49.311+0000 7f3cc3014040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-04T17:35:49.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.327+0000 7fad89e0d640 1 Processor -- start 2024-05-04T17:35:49.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.327+0000 7fad89e0d640 1 -- start start 2024-05-04T17:35:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad89e0d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad840ab350 con 0x7fad840b4150 2024-05-04T17:35:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad89e0d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad840a98d0 con 0x7fad84145f20 2024-05-04T17:35:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad89e0d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad840b2410 con 0x7fad840b4b10 2024-05-04T17:35:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad837fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fad840b4b10 0x7fad8413f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36066/0 (socket says 172.21.15.110:36066) 2024-05-04T17:35:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad837fe640 1 -- 172.21.15.110:0/1704661967 learned_addr learned my addr 172.21.15.110:0/1704661967 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad82ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fad840b4150 0x7fad84145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37648/0 (socket says 172.21.15.110:37648) 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363431623 0 0) 0x7fad840b2410 con 0x7fad840b4b10 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad6c003520 con 0x7fad840b4b10 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2276730361 0 0) 0x7fad840ab350 con 0x7fad840b4150 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad840b2410 con 0x7fad840b4150 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514335493 0 0) 0x7fad840a98d0 con 0x7fad84145f20 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.328+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad840ab350 con 0x7fad84145f20 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 754673358 0 0) 0x7fad6c003520 con 0x7fad840b4b10 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 >> v1:172.21.15.136:6789/0 conn(0x7fad84145f20 legacy=0x7fad8414dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 >> v1:172.21.15.110:6789/0 conn(0x7fad840b4150 legacy=0x7fad84145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad840a9df0 con 0x7fad840b4b10 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fad78001610 con 0x7fad840b4b10 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad78001930 con 0x7fad840b4b10 2024-05-04T17:35:49.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad827fc640 1 -- 172.21.15.110:0/1704661967 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fad78001d40 con 0x7fad840b4b10 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad89e0d640 1 -- 172.21.15.110:0/1704661967 >> v1:172.21.15.110:6790/0 conn(0x7fad840b4b10 legacy=0x7fad8413f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad89e0d640 1 -- 172.21.15.110:0/1704661967 shutdown_connections 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.329+0000 7fad89e0d640 1 -- 172.21.15.110:0/1704661967 >> 172.21.15.110:0/1704661967 conn(0x7fad84061c70 msgr2=0x7fad84062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.330+0000 7fad89e0d640 1 -- 172.21.15.110:0/1704661967 shutdown_connections 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.330+0000 7fad89e0d640 1 -- 172.21.15.110:0/1704661967 wait complete. 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.330+0000 7fad89e0d640 1 Processor -- start 2024-05-04T17:35:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad89e0d640 1 -- start start 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad89e0d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8413a190 con 0x7fad840b4b10 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad89e0d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8413ef70 con 0x7fad840b4150 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad89e0d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8413e5f0 con 0x7fad84145f20 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad82ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fad840b4b10 0x7fad841d7700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37654/0 (socket says 172.21.15.110:37654) 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad83fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fad84145f20 0x7fad841d7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36080/0 (socket says 172.21.15.110:36080) 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.331+0000 7fad82ffd640 1 -- 172.21.15.110:0/1115439190 learned_addr learned my addr 172.21.15.110:0/1115439190 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:49.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1970810434 0 0) 0x7fad8413ef70 con 0x7fad840b4150 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad70003520 con 0x7fad840b4150 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3799862803 0 0) 0x7fad8413e5f0 con 0x7fad84145f20 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad8413ef70 con 0x7fad84145f20 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008997517 0 0) 0x7fad8413a190 con 0x7fad840b4b10 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad8413e5f0 con 0x7fad840b4b10 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 316554266 0 0) 0x7fad70003520 con 0x7fad840b4150 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad8413a190 con 0x7fad840b4150 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fad780047e0 con 0x7fad840b4150 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1504811986 0 0) 0x7fad8413ef70 con 0x7fad84145f20 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad70003520 con 0x7fad84145f20 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fad7c001ad0 con 0x7fad84145f20 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2768291742 0 0) 0x7fad8413e5f0 con 0x7fad840b4b10 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad8413ef70 con 0x7fad840b4b10 2024-05-04T17:35:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fad74001280 con 0x7fad840b4b10 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1870200562 0 0) 0x7fad8413a190 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 >> v1:172.21.15.110:6790/0 conn(0x7fad84145f20 legacy=0x7fad841d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 >> v1:172.21.15.110:6789/0 conn(0x7fad840b4b10 legacy=0x7fad841d7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8413e7c0 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fad84139cb0 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fad8412dbc0 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad780011c0 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fad78004dd0 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fad78005340 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.332+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (unknown 2221240951 0 0) 0x7fad78005d00 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.333+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad64005080 con 0x7fad840b4150 2024-05-04T17:35:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.335+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fad78005fb0 con 0x7fad840b4150 2024-05-04T17:35:49.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.395+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad64005370 con 0x7fad840b4150 2024-05-04T17:35:49.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.396+0000 7fad6bfff640 1 -- 172.21.15.110:0/1115439190 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (unknown 4040575075 0 3859416838) 0x7fad78005630 con 0x7fad840b4150 2024-05-04T17:35:49.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:49.398 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"5cbb7704-6b22-42de-b82d-75ed642c3bca","created":"2024-05-04T17:35:47.067563+0000","modified":"2024-05-04T17:35:49.180095+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:35:49.180095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac107878-2a9c-435d-a24b-634939607eb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"895a609b-83a1-4d9d-a323-9e777df46326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"be5c569a-c157-48d0-b8d0-f44918cb1204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ff7aaeae-b017-4ece-8dab-2adea4056df6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"98a44aae-f478-4906-9f02-c63ae35f54d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c625ba28-d2ca-4ecb-8de2-4fecc934e626","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"db593666-b9e5-4cee-b022-f243b58345cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:35:49.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.397+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 >> v1:172.21.15.136:6789/0 conn(0x7fad840b4150 legacy=0x7fad8412d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:49.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.397+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 shutdown_connections 2024-05-04T17:35:49.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.397+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 >> 172.21.15.110:0/1115439190 conn(0x7fad84061c70 msgr2=0x7fad840b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:49.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.397+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 shutdown_connections 2024-05-04T17:35:49.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:49.397+0000 7fad89e0d640 1 -- 172.21.15.110:0/1115439190 wait complete. 2024-05-04T17:35:49.409 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-04T17:35:49.409 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-04T17:35:49.410 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:49.409+0000 7f28c840f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-04T17:35:49.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:49.574+0000 7f3cc3014040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-04T17:35:49.628 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:49.627+0000 7f3cc3014040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-04T17:35:49.668 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:49.667+0000 7f28c840f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-04T17:35:49.719 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:49.718+0000 7f28c840f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-04T17:35:49.742 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:49.741+0000 7f3cc3014040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-04T17:35:49.802 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-04T17:35:49.801+0000 7f27b1ddc640 -1 Falling back to public interface 2024-05-04T17:35:49.811 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-04T17:35:49.810+0000 7f32de014640 -1 Falling back to public interface 2024-05-04T17:35:49.816 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-04T17:35:49.815+0000 7f8c11405640 -1 Falling back to public interface 2024-05-04T17:35:49.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-05-04T17:35:49.821+0000 7f4d9c5a7640 -1 Falling back to public interface 2024-05-04T17:35:49.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-05-04T17:35:49.826+0000 7f5ef1e18640 -1 Falling back to public interface 2024-05-04T17:35:49.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-05-04T17:35:49.828+0000 7f7c27a24640 -1 Falling back to public interface 2024-05-04T17:35:49.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-05-04T17:35:49.828+0000 7fdfbb62e640 -1 Falling back to public interface 2024-05-04T17:35:49.836 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:49.835+0000 7f28c840f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-04T17:35:49.837 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-04T17:35:49.837+0000 7fdfee214640 -1 Falling back to public interface 2024-05-04T17:35:50.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.015+0000 7f3cc3014040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-04T17:35:50.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.062+0000 7f3cc3014040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-04T17:35:50.113 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.112+0000 7f28c840f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-04T17:35:50.161 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.160+0000 7f28c840f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-04T17:35:50.214 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:35:50.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.219+0000 7f3cc3014040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-04T17:35:50.304 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.302+0000 7fda3e40e640 1 Processor -- start 2024-05-04T17:35:50.304 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.302+0000 7fda3e40e640 1 -- start start 2024-05-04T17:35:50.306 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda3e40e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380b4860 con 0x7fda380a9d30 2024-05-04T17:35:50.306 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda3e40e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380b44c0 con 0x7fda380b1b80 2024-05-04T17:35:50.306 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda3e40e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda38145ff0 con 0x7fda380a9950 2024-05-04T17:35:50.306 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda37fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fda380a9950 0x7fda3813f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36086/0 (socket says 172.21.15.110:36086) 2024-05-04T17:35:50.306 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda37fff640 1 -- 172.21.15.110:0/2379838294 learned_addr learned my addr 172.21.15.110:0/2379838294 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:50.306 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269468958 0 0) 0x7fda380b4860 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda20003520 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980247768 0 0) 0x7fda38145ff0 con 0x7fda380a9950 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda380b4860 con 0x7fda380a9950 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247149538 0 0) 0x7fda380b44c0 con 0x7fda380b1b80 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda38145ff0 con 0x7fda380b1b80 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2185678419 0 0) 0x7fda20003520 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.303+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 >> v1:172.21.15.110:6790/0 conn(0x7fda380a9950 legacy=0x7fda3813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 >> v1:172.21.15.136:6789/0 conn(0x7fda380b1b80 legacy=0x7fda3814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda380b4d80 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fda28001540 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda28001860 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda36ffd640 1 -- 172.21.15.110:0/2379838294 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fda28001c30 con 0x7fda380a9d30 2024-05-04T17:35:50.307 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda3e40e640 1 -- 172.21.15.110:0/2379838294 >> v1:172.21.15.110:6789/0 conn(0x7fda380a9d30 legacy=0x7fda381458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:50.308 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda3e40e640 1 -- 172.21.15.110:0/2379838294 shutdown_connections 2024-05-04T17:35:50.308 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda3e40e640 1 -- 172.21.15.110:0/2379838294 >> 172.21.15.110:0/2379838294 conn(0x7fda3806d460 msgr2=0x7fda3806d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:50.308 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda3e40e640 1 -- 172.21.15.110:0/2379838294 shutdown_connections 2024-05-04T17:35:50.308 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.304+0000 7fda3e40e640 1 -- 172.21.15.110:0/2379838294 wait complete. 2024-05-04T17:35:50.308 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.305+0000 7fda3e40e640 1 Processor -- start 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.305+0000 7fda3e40e640 1 -- start start 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda3e40e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380ad5e0 con 0x7fda380b1b80 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda3e40e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380ad8c0 con 0x7fda380a9d30 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda3e40e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380adba0 con 0x7fda380a9950 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda37fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fda380a9950 0x7fda380ae2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36094/0 (socket says 172.21.15.110:36094) 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda3c9ab640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fda380b1b80 0x7fda38141d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37672/0 (socket says 172.21.15.110:37672) 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda37fff640 1 -- 172.21.15.110:0/1236667939 learned_addr learned my addr 172.21.15.110:0/1236667939 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485096337 0 0) 0x7fda380adba0 con 0x7fda380a9950 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda24003520 con 0x7fda380a9950 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890264568 0 0) 0x7fda380ad5e0 con 0x7fda380b1b80 2024-05-04T17:35:50.311 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda380adba0 con 0x7fda380b1b80 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3125816028 0 0) 0x7fda380ad8c0 con 0x7fda380a9d30 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda380ad5e0 con 0x7fda380a9d30 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2238982907 0 0) 0x7fda24003520 con 0x7fda380a9950 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda380ad8c0 con 0x7fda380a9950 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fda2c001870 con 0x7fda380a9950 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 213068201 0 0) 0x7fda380adba0 con 0x7fda380b1b80 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda24003520 con 0x7fda380b1b80 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.306+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fda30001b90 con 0x7fda380b1b80 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3529423623 0 0) 0x7fda380ad5e0 con 0x7fda380a9d30 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda380adba0 con 0x7fda380a9d30 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fda28001500 con 0x7fda380a9d30 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1940327637 0 0) 0x7fda380ad8c0 con 0x7fda380a9950 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 >> v1:172.21.15.136:6789/0 conn(0x7fda380a9d30 legacy=0x7fda380abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:50.312 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 >> v1:172.21.15.110:6789/0 conn(0x7fda380b1b80 legacy=0x7fda38141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda380ade80 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda380ac360 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fda381e38f0 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda2c0047e0 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fda2c004b30 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fda2c004ec0 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.307+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (unknown 2221240951 0 0) 0x7fda2c006a20 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.308+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda18005080 con 0x7fda380a9950 2024-05-04T17:35:50.313 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.310+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fda2c005800 con 0x7fda380a9950 2024-05-04T17:35:50.316 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.316+0000 7f28c840f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-04T17:35:50.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.361+0000 7f3cc3014040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-04T17:35:50.371 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.370+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda18005370 con 0x7fda380a9950 2024-05-04T17:35:50.372 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.371+0000 7fda34ff9640 1 -- 172.21.15.110:0/1236667939 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (unknown 4040575075 0 3859416838) 0x7fda2c005b80 con 0x7fda380a9950 2024-05-04T17:35:50.372 INFO:teuthology.misc.health.smithi110.stdout: 2024-05-04T17:35:50.372 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"5cbb7704-6b22-42de-b82d-75ed642c3bca","created":"2024-05-04T17:35:47.067563+0000","modified":"2024-05-04T17:35:49.180095+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:35:49.180095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac107878-2a9c-435d-a24b-634939607eb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"895a609b-83a1-4d9d-a323-9e777df46326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"be5c569a-c157-48d0-b8d0-f44918cb1204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ff7aaeae-b017-4ece-8dab-2adea4056df6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"98a44aae-f478-4906-9f02-c63ae35f54d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c625ba28-d2ca-4ecb-8de2-4fecc934e626","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"db593666-b9e5-4cee-b022-f243b58345cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:35:50.373 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.371+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 >> v1:172.21.15.110:6790/0 conn(0x7fda380a9950 legacy=0x7fda380ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:50.373 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.371+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 shutdown_connections 2024-05-04T17:35:50.373 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.371+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 >> 172.21.15.110:0/1236667939 conn(0x7fda3806d460 msgr2=0x7fda380b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:50.373 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.372+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 shutdown_connections 2024-05-04T17:35:50.373 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:50.372+0000 7fda3e40e640 1 -- 172.21.15.110:0/1236667939 wait complete. 2024-05-04T17:35:50.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-04T17:35:50.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.413+0000 7f3cc3014040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-04T17:35:50.455 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.454+0000 7f28c840f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-04T17:35:50.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.481+0000 7f3cc3014040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-04T17:35:50.503 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.503+0000 7f28c840f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-04T17:35:50.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:50.525+0000 7f3cc3014040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-04T17:35:50.572 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.571+0000 7f28c840f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-04T17:35:50.616 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:50.615+0000 7f28c840f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-04T17:35:51.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:51.001+0000 7f3cc3014040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-04T17:35:51.091 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:51.090+0000 7f28c840f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-04T17:35:51.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:51.117+0000 7f3cc3014040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-04T17:35:51.213 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-04T17:35:51.212+0000 7f28c840f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-04T17:35:52.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:52.129+0000 7f3c63941640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-04T17:35:52.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-05-04T17:35:52.129+0000 7f3c63941640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-04T17:35:53.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-05-04T17:35:53.289+0000 7f4d9c5a7640 -1 osd.0 0 log_to_monitors true 2024-05-04T17:35:53.314 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-04T17:35:53.311+0000 7f27b1ddc640 -1 osd.5 0 log_to_monitors true 2024-05-04T17:35:53.314 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-04T17:35:53.311+0000 7f8c11405640 -1 osd.7 0 log_to_monitors true 2024-05-04T17:35:53.344 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-04T17:35:53.342+0000 7f32de014640 -1 osd.4 0 log_to_monitors true 2024-05-04T17:35:53.357 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-04T17:35:53.356+0000 7fdfee214640 -1 osd.6 0 log_to_monitors true 2024-05-04T17:35:53.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-05-04T17:35:53.362+0000 7f7c27a24640 -1 osd.1 0 log_to_monitors true 2024-05-04T17:35:53.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-05-04T17:35:53.368+0000 7f5ef1e18640 -1 osd.2 0 log_to_monitors true 2024-05-04T17:35:53.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-05-04T17:35:53.368+0000 7fdfbb62e640 -1 osd.3 0 log_to_monitors true 2024-05-04T17:35:55.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-05-04T17:35:55.143+0000 7f4d97d1a640 -1 osd.0 0 waiting for initial osdmap 2024-05-04T17:35:55.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-05-04T17:35:55.144+0000 7f7c22ad3640 -1 osd.1 0 waiting for initial osdmap 2024-05-04T17:35:55.146 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-05-04T17:35:55.145+0000 7fdfb63a0640 -1 osd.3 0 waiting for initial osdmap 2024-05-04T17:35:55.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-05-04T17:35:55.151+0000 7f4d94354640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:55.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-05-04T17:35:55.151+0000 7f7c1f10d640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:55.155 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-04T17:35:55.154+0000 7fdfe945a640 -1 osd.6 0 waiting for initial osdmap 2024-05-04T17:35:55.155 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-05-04T17:35:55.154+0000 7fdfb29da640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:55.182 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-04T17:35:55.180+0000 7fdfe5a94640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:55.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-05-04T17:35:55.743+0000 7f5eed036640 -1 osd.2 0 waiting for initial osdmap 2024-05-04T17:35:55.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-05-04T17:35:55.748+0000 7f5ee5668640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:56.382 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-04T17:35:56.381+0000 7f8c0cb78640 -1 osd.7 0 waiting for initial osdmap 2024-05-04T17:35:56.389 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-04T17:35:56.388+0000 7f8c051aa640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:56.438 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-04T17:35:56.437+0000 7f27aed87640 -1 osd.5 0 waiting for initial osdmap 2024-05-04T17:35:56.445 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-04T17:35:56.444+0000 7f27a5b81640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:56.455 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-04T17:35:56.455+0000 7f32d925a640 -1 osd.4 0 waiting for initial osdmap 2024-05-04T17:35:56.460 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-04T17:35:56.460+0000 7f32d188c640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:35:57.191 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:35:57.284 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.283+0000 7f3fc0d4b640 1 Processor -- start 2024-05-04T17:35:57.284 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.283+0000 7f3fc0d4b640 1 -- start start 2024-05-04T17:35:57.286 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.283+0000 7f3fc0d4b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fbc0ac540 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.283+0000 7f3fc0d4b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fbc0ad1b0 con 0x7f3fbc149960 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.283+0000 7f3fc0d4b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fbc0ae020 con 0x7f3fbc0ae640 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb9d9b640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f3fbc0ae640 0x7f3fbc139c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46152/0 (socket says 172.21.15.110:46152) 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fba59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3fbc0abdd0 0x7f3fbc140800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35856/0 (socket says 172.21.15.110:35856) 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb9d9b640 1 -- 172.21.15.110:0/1538536911 learned_addr learned my addr 172.21.15.110:0/1538536911 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 769219848 0 0) 0x7f3fbc0ae020 con 0x7f3fbc0ae640 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fa8003520 con 0x7f3fbc0ae640 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195637233 0 0) 0x7f3fbc0ac540 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fbc0ae020 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1222466596 0 0) 0x7f3fbc0ad1b0 con 0x7f3fbc149960 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fbc0ac540 con 0x7f3fbc149960 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2431119085 0 0) 0x7f3fa8003520 con 0x7f3fbc0ae640 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 >> v1:172.21.15.136:6789/0 conn(0x7f3fbc149960 legacy=0x7f3fbc14bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.287 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 >> v1:172.21.15.110:6789/0 conn(0x7f3fbc0abdd0 legacy=0x7f3fbc140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fbc0ae300 con 0x7f3fbc0ae640 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3fa4001610 con 0x7f3fbc0ae640 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3fa4001870 con 0x7f3fbc0ae640 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.284+0000 7f3fb959a640 1 -- 172.21.15.110:0/1538536911 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3fa4001c00 con 0x7f3fbc0ae640 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.285+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/1538536911 >> v1:172.21.15.110:6790/0 conn(0x7f3fbc0ae640 legacy=0x7f3fbc139c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.285+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/1538536911 shutdown_connections 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.285+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/1538536911 >> 172.21.15.110:0/1538536911 conn(0x7f3fbc075520 msgr2=0x7f3fbc0b3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.285+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/1538536911 shutdown_connections 2024-05-04T17:35:57.288 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.285+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/1538536911 wait complete. 2024-05-04T17:35:57.291 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.286+0000 7f3fc0d4b640 1 Processor -- start 2024-05-04T17:35:57.291 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.286+0000 7f3fc0d4b640 1 -- start start 2024-05-04T17:35:57.291 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.286+0000 7f3fc0d4b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fbc141750 con 0x7f3fbc0ae640 2024-05-04T17:35:57.291 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.286+0000 7f3fc0d4b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fbc13c8d0 con 0x7f3fbc149960 2024-05-04T17:35:57.291 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.286+0000 7f3fc0d4b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fbc13d550 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb9d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3fbc0ae640 0x7f3fbc141b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35858/0 (socket says 172.21.15.110:35858) 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb9d9b640 1 -- 172.21.15.110:0/2267461218 learned_addr learned my addr 172.21.15.110:0/2267461218 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225115288 0 0) 0x7f3fbc13d550 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f9c003520 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368966613 0 0) 0x7f3fbc0ae300 con 0x7f3fbc0ae640 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fbc13d550 con 0x7f3fbc0ae640 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364345723 0 0) 0x7f3fbc13c8d0 con 0x7f3fbc149960 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fbc0ae300 con 0x7f3fbc149960 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3651992059 0 0) 0x7f3f9c003520 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fbc13c8d0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3fac001a10 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1534307248 0 0) 0x7f3fbc13d550 con 0x7f3fbc0ae640 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f9c003520 con 0x7f3fbc0ae640 2024-05-04T17:35:57.292 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3fa4001510 con 0x7f3fbc0ae640 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1894483914 0 0) 0x7f3fbc0ae300 con 0x7f3fbc149960 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fbc13d550 con 0x7f3fbc149960 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3fb4001b90 con 0x7f3fbc149960 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1094858632 0 0) 0x7f3fbc13c8d0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 >> v1:172.21.15.136:6789/0 conn(0x7f3fbc149960 legacy=0x7f3fbc1dffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 >> v1:172.21.15.110:6789/0 conn(0x7f3fbc0ae640 legacy=0x7f3fbc141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fbc143d30 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3fbc133db0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3fbc142460 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3fac001cc0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.287+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3fac004c30 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.288+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fbc133fe0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.289+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) ==== 90407+0+0 (unknown 2763682855 0 0) 0x7f3fac01ad60 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.293 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.289+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(14..14 src has 1..14) ==== 3726+0+0 (unknown 1184848397 0 0) 0x7f3fac085f00 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.294 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.293+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f3fac054fe0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.470 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.469+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3fbc04fe40 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.471 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.470+0000 7f3fb2ffd640 1 -- 172.21.15.110:0/2267461218 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10919 (unknown 92231148 0 1918486198) 0x7f3fac0548e0 con 0x7f3fbc0abdd0 2024-05-04T17:35:57.471 INFO:teuthology.misc.health.smithi110.stdout: 2024-05-04T17:35:57.471 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":14,"fsid":"5cbb7704-6b22-42de-b82d-75ed642c3bca","created":"2024-05-04T17:35:47.067563+0000","modified":"2024-05-04T17:35:57.156954+0000","last_up_change":"2024-05-04T17:35:57.156954+0000","last_in_change":"2024-05-04T17:35:49.180095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:35:57.156872+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac107878-2a9c-435d-a24b-634939607eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6800","nonce":1712129713}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6801","nonce":1712129713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6803","nonce":1712129713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6802","nonce":1712129713}]},"public_addr":"172.21.15.110:6800/1712129713","cluster_addr":"172.21.15.110:6801/1712129713","heartbeat_back_addr":"172.21.15.110:6803/1712129713","heartbeat_front_addr":"172.21.15.110:6802/1712129713","state":["exists","up"]},{"osd":1,"uuid":"895a609b-83a1-4d9d-a323-9e777df46326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6808","nonce":1520708426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6810","nonce":1520708426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6814","nonce":1520708426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6812","nonce":1520708426}]},"public_addr":"172.21.15.110:6808/1520708426","cluster_addr":"172.21.15.110:6810/1520708426","heartbeat_back_addr":"172.21.15.110:6814/1520708426","heartbeat_front_addr":"172.21.15.110:6812/1520708426","state":["exists","up"]},{"osd":2,"uuid":"be5c569a-c157-48d0-b8d0-f44918cb1204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6804","nonce":4241686922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6805","nonce":4241686922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6807","nonce":4241686922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6806","nonce":4241686922}]},"public_addr":"172.21.15.110:6804/4241686922","cluster_addr":"172.21.15.110:6805/4241686922","heartbeat_back_addr":"172.21.15.110:6807/4241686922","heartbeat_front_addr":"172.21.15.110:6806/4241686922","state":["exists","up"]},{"osd":3,"uuid":"ff7aaeae-b017-4ece-8dab-2adea4056df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6809","nonce":2632237368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6811","nonce":2632237368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6815","nonce":2632237368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6813","nonce":2632237368}]},"public_addr":"172.21.15.110:6809/2632237368","cluster_addr":"172.21.15.110:6811/2632237368","heartbeat_back_addr":"172.21.15.110:6815/2632237368","heartbeat_front_addr":"172.21.15.110:6813/2632237368","state":["exists","up"]},{"osd":4,"uuid":"98a44aae-f478-4906-9f02-c63ae35f54d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6804","nonce":2025414972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6805","nonce":2025414972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6807","nonce":2025414972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6806","nonce":2025414972}]},"public_addr":"172.21.15.136:6804/2025414972","cluster_addr":"172.21.15.136:6805/2025414972","heartbeat_back_addr":"172.21.15.136:6807/2025414972","heartbeat_front_addr":"172.21.15.136:6806/2025414972","state":["exists","up"]},{"osd":5,"uuid":"c625ba28-d2ca-4ecb-8de2-4fecc934e626","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6800","nonce":2872378682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6801","nonce":2872378682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6803","nonce":2872378682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6802","nonce":2872378682}]},"public_addr":"172.21.15.136:6800/2872378682","cluster_addr":"172.21.15.136:6801/2872378682","heartbeat_back_addr":"172.21.15.136:6803/2872378682","heartbeat_front_addr":"172.21.15.136:6802/2872378682","state":["exists","up"]},{"osd":6,"uuid":"ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6812","nonce":239937373}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6813","nonce":239937373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6815","nonce":239937373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6814","nonce":239937373}]},"public_addr":"172.21.15.136:6812/239937373","cluster_addr":"172.21.15.136:6813/239937373","heartbeat_back_addr":"172.21.15.136:6815/239937373","heartbeat_front_addr":"172.21.15.136:6814/239937373","state":["exists","up"]},{"osd":7,"uuid":"db593666-b9e5-4cee-b022-f243b58345cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6808","nonce":2884573928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6809","nonce":2884573928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6811","nonce":2884573928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6810","nonce":2884573928}]},"public_addr":"172.21.15.136:6808/2884573928","cluster_addr":"172.21.15.136:6809/2884573928","heartbeat_back_addr":"172.21.15.136:6811/2884573928","heartbeat_front_addr":"172.21.15.136:6810/2884573928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.320324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.405374+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.374617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.391393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.375307+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:35:57.473 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.472+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 >> v1:172.21.15.110:6816/783516658 conn(0x7f3f9c06c9d0 legacy=0x7f3f9c06ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.473 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.472+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 >> v1:172.21.15.110:6790/0 conn(0x7f3fbc0abdd0 legacy=0x7f3fbc13c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.473 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.472+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 shutdown_connections 2024-05-04T17:35:57.473 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.472+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 >> 172.21.15.110:0/2267461218 conn(0x7f3fbc075520 msgr2=0x7f3fbc0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:57.473 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.472+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 shutdown_connections 2024-05-04T17:35:57.473 INFO:teuthology.misc.health.smithi110.stderr:2024-05-04T17:35:57.472+0000 7f3fc0d4b640 1 -- 172.21.15.110:0/2267461218 wait complete. 2024-05-04T17:35:57.483 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-04T17:35:57.483 INFO:tasks.ceph:Creating RBD pool 2024-05-04T17:35:57.483 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-04T17:35:57.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.577+0000 7f029d4df640 1 Processor -- start 2024-05-04T17:35:57.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.577+0000 7f029d4df640 1 -- start start 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f029d4df640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02980ab350 con 0x7f02980b4150 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f029d4df640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02980a98d0 con 0x7f0298145f20 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f029d4df640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02980b2410 con 0x7f02980b4b10 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f02967fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f02980b4150 0x7f0298145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35866/0 (socket says 172.21.15.110:35866) 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f02967fc640 1 -- 172.21.15.110:0/1175270483 learned_addr learned my addr 172.21.15.110:0/1175270483 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1950311825 0 0) 0x7f02980b2410 con 0x7f02980b4b10 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0284003520 con 0x7f02980b4b10 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831108404 0 0) 0x7f02980ab350 con 0x7f02980b4150 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02980b2410 con 0x7f02980b4150 2024-05-04T17:35:57.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2856609636 0 0) 0x7f02980a98d0 con 0x7f0298145f20 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02980ab350 con 0x7f0298145f20 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.578+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3965285869 0 0) 0x7f0284003520 con 0x7f02980b4b10 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 >> v1:172.21.15.136:6789/0 conn(0x7f0298145f20 legacy=0x7f029814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 >> v1:172.21.15.110:6789/0 conn(0x7f02980b4150 legacy=0x7f0298145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02980a9df0 con 0x7f02980b4b10 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0288001990 con 0x7f02980b4b10 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0288001cb0 con 0x7f02980b4b10 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f0295ffb640 1 -- 172.21.15.110:0/1175270483 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f02880020c0 con 0x7f02980b4b10 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f029d4df640 1 -- 172.21.15.110:0/1175270483 >> v1:172.21.15.110:6790/0 conn(0x7f02980b4b10 legacy=0x7f029813f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f029d4df640 1 -- 172.21.15.110:0/1175270483 shutdown_connections 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f029d4df640 1 -- 172.21.15.110:0/1175270483 >> 172.21.15.110:0/1175270483 conn(0x7f0298061c70 msgr2=0x7f0298062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f029d4df640 1 -- 172.21.15.110:0/1175270483 shutdown_connections 2024-05-04T17:35:57.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.579+0000 7f029d4df640 1 -- 172.21.15.110:0/1175270483 wait complete. 2024-05-04T17:35:57.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.580+0000 7f029d4df640 1 Processor -- start 2024-05-04T17:35:57.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.580+0000 7f029d4df640 1 -- start start 2024-05-04T17:35:57.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f029d4df640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0298141890 con 0x7f02980b4150 2024-05-04T17:35:57.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f029d4df640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0298138fb0 con 0x7f0298145f20 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f029d4df640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0298141b70 con 0x7f02980b4b10 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f0296ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f02980b4150 0x7f029813e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35878/0 (socket says 172.21.15.110:35878) 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f0296ffd640 1 -- 172.21.15.110:0/2567318232 learned_addr learned my addr 172.21.15.110:0/2567318232 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926151265 0 0) 0x7f0298141b70 con 0x7f02980b4b10 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0278003520 con 0x7f02980b4b10 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546042335 0 0) 0x7f02980a9df0 con 0x7f02980b4150 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0298141b70 con 0x7f02980b4150 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2019086648 0 0) 0x7f0278003520 con 0x7f02980b4b10 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02980a9df0 con 0x7f02980b4b10 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f02800019b0 con 0x7f02980b4b10 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2735425803 0 0) 0x7f0298141b70 con 0x7f02980b4150 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0278003520 con 0x7f02980b4150 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0288001950 con 0x7f02980b4150 2024-05-04T17:35:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152101804 0 0) 0x7f0298138fb0 con 0x7f0298145f20 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.581+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0298141b70 con 0x7f0298145f20 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 366014514 0 0) 0x7f02980a9df0 con 0x7f02980b4b10 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 >> v1:172.21.15.136:6789/0 conn(0x7f0298145f20 legacy=0x7f02981e2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 >> v1:172.21.15.110:6789/0 conn(0x7f02980b4150 legacy=0x7f029813e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0298141e50 con 0x7f02980b4b10 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f029813efc0 con 0x7f02980b4b10 2024-05-04T17:35:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0298140ce0 con 0x7f02980b4b10 2024-05-04T17:35:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02800047e0 con 0x7f02980b4b10 2024-05-04T17:35:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.582+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0280004b30 con 0x7f02980b4b10 2024-05-04T17:35:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.583+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f027c005080 con 0x7f02980b4b10 2024-05-04T17:35:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.583+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) ==== 90407+0+0 (unknown 2763682855 0 0) 0x7f028001ae00 con 0x7f02980b4b10 2024-05-04T17:35:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.583+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(14..14 src has 1..14) ==== 3726+0+0 (unknown 1184848397 0 0) 0x7f02800860d0 con 0x7f02980b4b10 2024-05-04T17:35:57.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.587+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f02800551b0 con 0x7f02980b4b10 2024-05-04T17:35:57.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:57.760+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f027c005b80 con 0x7f02980b4b10 2024-05-04T17:35:58.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.165+0000 7f028f7fe640 1 -- 172.21.15.110:0/2567318232 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) ==== 109+0+0 (unknown 1237045831 0 0) 0x7f0280054ab0 con 0x7f02980b4b10 2024-05-04T17:35:58.166 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2024-05-04T17:35:58.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.168+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 >> v1:172.21.15.110:6816/783516658 conn(0x7f027806c910 legacy=0x7f027806edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:58.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.168+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 >> v1:172.21.15.110:6790/0 conn(0x7f02980b4b10 legacy=0x7f02981df370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:58.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.169+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 shutdown_connections 2024-05-04T17:35:58.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.169+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 >> 172.21.15.110:0/2567318232 conn(0x7f0298061c70 msgr2=0x7f02980b7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:58.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.169+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 shutdown_connections 2024-05-04T17:35:58.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.169+0000 7f029d4df640 1 -- 172.21.15.110:0/2567318232 wait complete. 2024-05-04T17:35:58.183 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-04T17:35:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.288+0000 7f04fc66d640 1 Processor -- start 2024-05-04T17:35:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.288+0000 7f04fc66d640 1 -- start start 2024-05-04T17:35:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.288+0000 7f04fc66d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f40afea0 con 0x7f04f413c260 2024-05-04T17:35:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.288+0000 7f04fc66d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f4072660 con 0x7f04f40ae6e0 2024-05-04T17:35:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.288+0000 7f04fc66d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f413c670 con 0x7f04f40abe00 2024-05-04T17:35:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04fac0a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 0x7f04f414a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35904/0 (socket says 172.21.15.110:35904) 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9c08640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f04f40abe00 0x7f04f40ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46210/0 (socket says 172.21.15.110:46210) 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04fac0a640 1 -- 172.21.15.110:0/3844363684 learned_addr learned my addr 172.21.15.110:0/3844363684 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583058106 0 0) 0x7f04f40afea0 con 0x7f04f413c260 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04e0003520 con 0x7f04f413c260 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3027403828 0 0) 0x7f04f413c670 con 0x7f04f40abe00 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f40afea0 con 0x7f04f40abe00 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034815023 0 0) 0x7f04f4072660 con 0x7f04f40ae6e0 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f413c670 con 0x7f04f40ae6e0 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 173941921 0 0) 0x7f04f413c670 con 0x7f04f40ae6e0 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 >> v1:172.21.15.110:6790/0 conn(0x7f04f40abe00 legacy=0x7f04f40ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 legacy=0x7f04f414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f413c840 con 0x7f04f40ae6e0 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04f0001530 con 0x7f04f40ae6e0 2024-05-04T17:35:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04f0001760 con 0x7f04f40ae6e0 2024-05-04T17:35:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.289+0000 7f04f9407640 1 -- 172.21.15.110:0/3844363684 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04f0001bb0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.290+0000 7f04fc66d640 1 -- 172.21.15.110:0/3844363684 >> v1:172.21.15.136:6789/0 conn(0x7f04f40ae6e0 legacy=0x7f04f40aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.290+0000 7f04fc66d640 1 -- 172.21.15.110:0/3844363684 shutdown_connections 2024-05-04T17:35:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.290+0000 7f04fc66d640 1 -- 172.21.15.110:0/3844363684 >> 172.21.15.110:0/3844363684 conn(0x7f04f40b3c50 msgr2=0x7f04f40b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.290+0000 7f04fc66d640 1 -- 172.21.15.110:0/3844363684 shutdown_connections 2024-05-04T17:35:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.290+0000 7f04fc66d640 1 -- 172.21.15.110:0/3844363684 wait complete. 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.291+0000 7f04fc66d640 1 Processor -- start 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.291+0000 7f04fc66d640 1 -- start start 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fc66d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f413e970 con 0x7f04f413c260 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fc66d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f413d6b0 con 0x7f04f40abe00 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fc66d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f41d6f80 con 0x7f04f40ae6e0 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04f9c08640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f04f40ae6e0 0x7f04f41d6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46226/0 (socket says 172.21.15.110:46226) 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04f9c08640 1 -- 172.21.15.110:0/3901739292 learned_addr learned my addr 172.21.15.110:0/3901739292 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fac0a640 1 --1- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 0x7f04f413f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35916/0 (socket says 172.21.15.110:35916) 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327641219 0 0) 0x7f04f413e970 con 0x7f04f413c260 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04d8003520 con 0x7f04f413c260 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942940076 0 0) 0x7f04f41d6f80 con 0x7f04f40ae6e0 2024-05-04T17:35:58.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f413e970 con 0x7f04f40ae6e0 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fac0a640 1 -- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 legacy=0x7f04f413f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fac0a640 1 -- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 legacy=0x7f04f413f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fac0a640 1 --1- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 0x7f04f413f580 :-1 s=OPENED pgs=76 cs=1 l=1).handle_message read tag failed 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fac0a640 1 --1- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 0x7f04f413f580 :-1 s=OPENED pgs=76 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462697446 0 0) 0x7f04f413d6b0 con 0x7f04f40abe00 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f41d6f80 con 0x7f04f40abe00 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2699972151 0 0) 0x7f04f413e970 con 0x7f04f40ae6e0 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04f413d6b0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04e40017f0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4122393206 0 0) 0x7f04f41d6f80 con 0x7f04f40abe00 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04f413e970 con 0x7f04f40abe00 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04f0001b70 con 0x7f04f40abe00 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2846723923 0 0) 0x7f04f413d6b0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 >> v1:172.21.15.136:6789/0 conn(0x7f04f40abe00 legacy=0x7f04f413cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6789/0 conn(0x7f04f413c260 legacy=0x7f04f413f580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f41d7150 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04f41430f0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f04f413ef70 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.292+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04e40047e0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.293+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04e4004b30 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.294+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04dc005080 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.294+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) ==== 90407+0+0 (unknown 2763682855 0 0) 0x7f04e401ae00 con 0x7f04f40ae6e0 2024-05-04T17:35:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.294+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 4079+0+0 (unknown 1659019351 0 0) 0x7f04e4086130 con 0x7f04f40ae6e0 2024-05-04T17:35:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.298+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f04e40550b0 con 0x7f04f40ae6e0 2024-05-04T17:35:58.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:58.490+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f04dc005370 con 0x7f04f40ae6e0 2024-05-04T17:35:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.216+0000 7f04eaffd640 1 -- 172.21.15.110:0/3901739292 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) ==== 173+0+0 (unknown 4195546702 0 0) 0x7f04e40549b0 con 0x7f04f40ae6e0 2024-05-04T17:35:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-04T17:35:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.220+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6816/783516658 conn(0x7f04d806c890 legacy=0x7f04d806ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.220+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 >> v1:172.21.15.110:6790/0 conn(0x7f04f40ae6e0 legacy=0x7f04f41d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.220+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 shutdown_connections 2024-05-04T17:35:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.220+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 >> 172.21.15.110:0/3901739292 conn(0x7f04f40b3c50 msgr2=0x7f04f4135c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.220+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 shutdown_connections 2024-05-04T17:35:59.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.220+0000 7f04fc66d640 1 -- 172.21.15.110:0/3901739292 wait complete. 2024-05-04T17:35:59.233 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-04T17:35:59.234 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-04T17:35:59.234 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-04T17:35:59.234 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-04T17:35:59.235 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-04T17:35:59.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf35700640 1 Processor -- start 2024-05-04T17:35:59.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf35700640 1 -- start start 2024-05-04T17:35:59.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf35700640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf30072830 con 0x7fbf300b0600 2024-05-04T17:35:59.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf35700640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf30072aa0 con 0x7fbf30070850 2024-05-04T17:35:59.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf35700640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf30140540 con 0x7fbf300afcb0 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf27fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbf300b0600 0x7fbf30140a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35934/0 (socket says 172.21.15.110:35934) 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.330+0000 7fbf2f7fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fbf300afcb0 0x7fbf3014b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46246/0 (socket says 172.21.15.110:46246) 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf27fff640 1 -- 172.21.15.110:0/457352166 learned_addr learned my addr 172.21.15.110:0/457352166 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 45835985 0 0) 0x7fbf30140540 con 0x7fbf300afcb0 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf1c003520 con 0x7fbf300afcb0 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502080821 0 0) 0x7fbf30072830 con 0x7fbf300b0600 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf30140540 con 0x7fbf300b0600 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230273004 0 0) 0x7fbf20001110 con 0x7fbf30070850 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf30072830 con 0x7fbf30070850 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3474320467 0 0) 0x7fbf30140540 con 0x7fbf300b0600 2024-05-04T17:35:59.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 >> v1:172.21.15.110:6790/0 conn(0x7fbf300afcb0 legacy=0x7fbf3014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 >> v1:172.21.15.136:6789/0 conn(0x7fbf30070850 legacy=0x7fbf30070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf30072d80 con 0x7fbf300b0600 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbf18001520 con 0x7fbf300b0600 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf18001790 con 0x7fbf300b0600 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.331+0000 7fbf2e7fc640 1 -- 172.21.15.110:0/457352166 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbf18001ba0 con 0x7fbf300b0600 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.332+0000 7fbf35700640 1 -- 172.21.15.110:0/457352166 >> v1:172.21.15.110:6789/0 conn(0x7fbf300b0600 legacy=0x7fbf30140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.332+0000 7fbf35700640 1 -- 172.21.15.110:0/457352166 shutdown_connections 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.332+0000 7fbf35700640 1 -- 172.21.15.110:0/457352166 >> 172.21.15.110:0/457352166 conn(0x7fbf300b5c00 msgr2=0x7fbf300b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.332+0000 7fbf35700640 1 -- 172.21.15.110:0/457352166 shutdown_connections 2024-05-04T17:35:59.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.332+0000 7fbf35700640 1 -- 172.21.15.110:0/457352166 wait complete. 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf35700640 1 Processor -- start 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf35700640 1 -- start start 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf35700640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf300ab650 con 0x7fbf300b0600 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf35700640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf300ac210 con 0x7fbf30070850 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf35700640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf300ac4f0 con 0x7fbf300afcb0 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf27fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fbf300afcb0 0x7fbf301d5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46260/0 (socket says 172.21.15.110:46260) 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf27fff640 1 -- 172.21.15.110:0/3881981293 learned_addr learned my addr 172.21.15.110:0/3881981293 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.333+0000 7fbf2f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbf300b0600 0x7fbf30133e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35938/0 (socket says 172.21.15.110:35938) 2024-05-04T17:35:59.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900556758 0 0) 0x7fbf300ac4f0 con 0x7fbf300afcb0 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf10003520 con 0x7fbf300afcb0 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827439471 0 0) 0x7fbf300ab650 con 0x7fbf300b0600 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf300ac4f0 con 0x7fbf300b0600 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547851300 0 0) 0x7fbf300ac210 con 0x7fbf30070850 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf300ab650 con 0x7fbf30070850 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 701768748 0 0) 0x7fbf10003520 con 0x7fbf300afcb0 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf300ac210 con 0x7fbf300afcb0 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1546037694 0 0) 0x7fbf300ac4f0 con 0x7fbf300b0600 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf10003520 con 0x7fbf300b0600 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbf18001470 con 0x7fbf300afcb0 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbf28001490 con 0x7fbf300b0600 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 845592178 0 0) 0x7fbf300ab650 con 0x7fbf30070850 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf300ac4f0 con 0x7fbf30070850 2024-05-04T17:35:59.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbf20001a40 con 0x7fbf30070850 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 178342728 0 0) 0x7fbf300ac210 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 >> v1:172.21.15.136:6789/0 conn(0x7fbf30070850 legacy=0x7fbf300aaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 >> v1:172.21.15.110:6789/0 conn(0x7fbf300b0600 legacy=0x7fbf30133e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf300aa210 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf301349e0 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf3013bb60 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf18004a90 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.334+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbf18004c40 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.335+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf14005080 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.336+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) ==== 90407+0+0 (unknown 2763682855 0 0) 0x7fbf18004de0 con 0x7fbf300afcb0 2024-05-04T17:35:59.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.336+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 4090+0+0 (unknown 1923690868 0 0) 0x7fbf180860e0 con 0x7fbf300afcb0 2024-05-04T17:35:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.341+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fbf180550b0 con 0x7fbf300afcb0 2024-05-04T17:35:59.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.562+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbf14005370 con 0x7fbf300afcb0 2024-05-04T17:35:59.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.565+0000 7fbf277fe640 1 -- 172.21.15.110:0/3881981293 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+172789 (unknown 3797009233 0 2462539820) 0x7fbf180549b0 con 0x7fbf300afcb0 2024-05-04T17:35:59.569 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:59.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.567+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 >> v1:172.21.15.110:6816/783516658 conn(0x7fbf1006ca00 legacy=0x7fbf1006eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.567+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 >> v1:172.21.15.110:6790/0 conn(0x7fbf300afcb0 legacy=0x7fbf301d5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.567+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 shutdown_connections 2024-05-04T17:35:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.567+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 >> 172.21.15.110:0/3881981293 conn(0x7fbf300b5c00 msgr2=0x7fbf300b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.567+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 shutdown_connections 2024-05-04T17:35:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.568+0000 7fbf35700640 1 -- 172.21.15.110:0/3881981293 wait complete. 2024-05-04T17:35:59.579 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":4,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6816","nonce":783516658}]},"active_addr":"172.21.15.110:6816/783516658","active_change":"2024-05-04T17:35:51.119216+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1107922850}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":3148020125}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1519384262}]}]} 2024-05-04T17:35:59.585 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-04T17:35:59.585 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-04T17:35:59.585 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:35:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.681+0000 7f73a8ba0640 1 Processor -- start 2024-05-04T17:35:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.681+0000 7f73a8ba0640 1 -- start start 2024-05-04T17:35:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.681+0000 7f73a8ba0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a40b4430 con 0x7f73a40b39f0 2024-05-04T17:35:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.681+0000 7f73a8ba0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a40aae40 con 0x7f73a40726b0 2024-05-04T17:35:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.681+0000 7f73a8ba0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a40ab120 con 0x7f73a4142780 2024-05-04T17:35:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f739bfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f73a40b39f0 0x7f73a4142070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35952/0 (socket says 172.21.15.110:35952) 2024-05-04T17:35:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f739bfff640 1 -- 172.21.15.110:0/1106856514 learned_addr learned my addr 172.21.15.110:0/1106856514 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3389777809 0 0) 0x7f73a40ab120 con 0x7f73a4142780 2024-05-04T17:35:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7390003520 con 0x7f73a4142780 2024-05-04T17:35:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500519757 0 0) 0x7f73a40b4430 con 0x7f73a40b39f0 2024-05-04T17:35:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a40ab120 con 0x7f73a40b39f0 2024-05-04T17:35:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1572109670 0 0) 0x7f7390003520 con 0x7f73a4142780 2024-05-04T17:35:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 >> v1:172.21.15.136:6789/0 conn(0x7f73a40726b0 legacy=0x7f73a413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 >> v1:172.21.15.110:6789/0 conn(0x7f73a40b39f0 legacy=0x7f73a4142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a40a9a20 con 0x7f73a4142780 2024-05-04T17:35:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f739c001990 con 0x7f73a4142780 2024-05-04T17:35:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f739c001c40 con 0x7f73a4142780 2024-05-04T17:35:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.682+0000 7f73a2ffd640 1 -- 172.21.15.110:0/1106856514 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f739c002050 con 0x7f73a4142780 2024-05-04T17:35:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.683+0000 7f73a8ba0640 1 -- 172.21.15.110:0/1106856514 >> v1:172.21.15.110:6790/0 conn(0x7f73a4142780 legacy=0x7f73a414d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.683+0000 7f73a8ba0640 1 -- 172.21.15.110:0/1106856514 shutdown_connections 2024-05-04T17:35:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.683+0000 7f73a8ba0640 1 -- 172.21.15.110:0/1106856514 >> 172.21.15.110:0/1106856514 conn(0x7f73a40b7b20 msgr2=0x7f73a40b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:59.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.683+0000 7f73a8ba0640 1 -- 172.21.15.110:0/1106856514 shutdown_connections 2024-05-04T17:35:59.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.683+0000 7f73a8ba0640 1 -- 172.21.15.110:0/1106856514 wait complete. 2024-05-04T17:35:59.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a8ba0640 1 Processor -- start 2024-05-04T17:35:59.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a8ba0640 1 -- start start 2024-05-04T17:35:59.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a8ba0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a4144560 con 0x7f73a4142780 2024-05-04T17:35:59.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a8ba0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a41415c0 con 0x7f73a40b39f0 2024-05-04T17:35:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a8ba0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a413d680 con 0x7f73a40726b0 2024-05-04T17:35:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a37fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f73a40726b0 0x7f73a4141ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46276/0 (socket says 172.21.15.110:46276) 2024-05-04T17:35:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a37fe640 1 -- 172.21.15.110:0/657488189 learned_addr learned my addr 172.21.15.110:0/657488189 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894943102 0 0) 0x7f73a4144560 con 0x7f73a4142780 2024-05-04T17:35:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7384003520 con 0x7f73a4142780 2024-05-04T17:35:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 56047453 0 0) 0x7f73a413d680 con 0x7f73a40726b0 2024-05-04T17:35:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.684+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a4144560 con 0x7f73a40726b0 2024-05-04T17:35:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1843697374 0 0) 0x7f73a41415c0 con 0x7f73a40b39f0 2024-05-04T17:35:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a413d680 con 0x7f73a40b39f0 2024-05-04T17:35:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1779014910 0 0) 0x7f73a4144560 con 0x7f73a40726b0 2024-05-04T17:35:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73a41415c0 con 0x7f73a40726b0 2024-05-04T17:35:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1289815826 0 0) 0x7f7384003520 con 0x7f73a4142780 2024-05-04T17:35:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73a4144560 con 0x7f73a4142780 2024-05-04T17:35:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7394001910 con 0x7f73a40726b0 2024-05-04T17:35:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f739c001df0 con 0x7f73a4142780 2024-05-04T17:35:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3453740840 0 0) 0x7f73a41415c0 con 0x7f73a40726b0 2024-05-04T17:35:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 >> v1:172.21.15.136:6789/0 conn(0x7f73a40b39f0 legacy=0x7f73a4140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 >> v1:172.21.15.110:6789/0 conn(0x7f73a4142780 legacy=0x7f73a41e19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a4143370 con 0x7f73a40726b0 2024-05-04T17:35:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73a4145990 con 0x7f73a40726b0 2024-05-04T17:35:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f73a41e32a0 con 0x7f73a40726b0 2024-05-04T17:35:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7394001ce0 con 0x7f73a40726b0 2024-05-04T17:35:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.685+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7394003a60 con 0x7f73a40726b0 2024-05-04T17:35:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.686+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7388005080 con 0x7f73a40726b0 2024-05-04T17:35:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.686+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) ==== 90407+0+0 (unknown 2763682855 0 0) 0x7f7394003a60 con 0x7f73a40726b0 2024-05-04T17:35:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.687+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 4090+0+0 (unknown 1923690868 0 0) 0x7f7394085fb0 con 0x7f73a40726b0 2024-05-04T17:35:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.691+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f7394054f20 con 0x7f73a40726b0 2024-05-04T17:35:59.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.799+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f73940504e0 con 0x7f73a40726b0 2024-05-04T17:35:59.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.868+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7388005370 con 0x7f73a40726b0 2024-05-04T17:35:59.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.869+0000 7f73a0ff9640 1 -- 172.21.15.110:0/657488189 <== mon.2 v1:172.21.15.110:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12816 (unknown 1054498076 0 2811290940) 0x7f739405b3c0 con 0x7f73a40726b0 2024-05-04T17:35:59.873 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:35:59.873 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"fsid":"5cbb7704-6b22-42de-b82d-75ed642c3bca","created":"2024-05-04T17:35:47.067563+0000","modified":"2024-05-04T17:35:59.164344+0000","last_up_change":"2024-05-04T17:35:57.156954+0000","last_in_change":"2024-05-04T17:35:49.180095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:35:57.156872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-04T17:35:57.779672+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac107878-2a9c-435d-a24b-634939607eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6800","nonce":1712129713}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6801","nonce":1712129713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6803","nonce":1712129713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6802","nonce":1712129713}]},"public_addr":"172.21.15.110:6800/1712129713","cluster_addr":"172.21.15.110:6801/1712129713","heartbeat_back_addr":"172.21.15.110:6803/1712129713","heartbeat_front_addr":"172.21.15.110:6802/1712129713","state":["exists","up"]},{"osd":1,"uuid":"895a609b-83a1-4d9d-a323-9e777df46326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6808","nonce":1520708426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6810","nonce":1520708426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6814","nonce":1520708426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6812","nonce":1520708426}]},"public_addr":"172.21.15.110:6808/1520708426","cluster_addr":"172.21.15.110:6810/1520708426","heartbeat_back_addr":"172.21.15.110:6814/1520708426","heartbeat_front_addr":"172.21.15.110:6812/1520708426","state":["exists","up"]},{"osd":2,"uuid":"be5c569a-c157-48d0-b8d0-f44918cb1204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6804","nonce":4241686922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6805","nonce":4241686922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6807","nonce":4241686922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6806","nonce":4241686922}]},"public_addr":"172.21.15.110:6804/4241686922","cluster_addr":"172.21.15.110:6805/4241686922","heartbeat_back_addr":"172.21.15.110:6807/4241686922","heartbeat_front_addr":"172.21.15.110:6806/4241686922","state":["exists","up"]},{"osd":3,"uuid":"ff7aaeae-b017-4ece-8dab-2adea4056df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6809","nonce":2632237368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6811","nonce":2632237368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6815","nonce":2632237368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6813","nonce":2632237368}]},"public_addr":"172.21.15.110:6809/2632237368","cluster_addr":"172.21.15.110:6811/2632237368","heartbeat_back_addr":"172.21.15.110:6815/2632237368","heartbeat_front_addr":"172.21.15.110:6813/2632237368","state":["exists","up"]},{"osd":4,"uuid":"98a44aae-f478-4906-9f02-c63ae35f54d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6804","nonce":2025414972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6805","nonce":2025414972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6807","nonce":2025414972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6806","nonce":2025414972}]},"public_addr":"172.21.15.136:6804/2025414972","cluster_addr":"172.21.15.136:6805/2025414972","heartbeat_back_addr":"172.21.15.136:6807/2025414972","heartbeat_front_addr":"172.21.15.136:6806/2025414972","state":["exists","up"]},{"osd":5,"uuid":"c625ba28-d2ca-4ecb-8de2-4fecc934e626","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6800","nonce":2872378682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6801","nonce":2872378682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6803","nonce":2872378682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6802","nonce":2872378682}]},"public_addr":"172.21.15.136:6800/2872378682","cluster_addr":"172.21.15.136:6801/2872378682","heartbeat_back_addr":"172.21.15.136:6803/2872378682","heartbeat_front_addr":"172.21.15.136:6802/2872378682","state":["exists","up"]},{"osd":6,"uuid":"ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6812","nonce":239937373}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6813","nonce":239937373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6815","nonce":239937373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6814","nonce":239937373}]},"public_addr":"172.21.15.136:6812/239937373","cluster_addr":"172.21.15.136:6813/239937373","heartbeat_back_addr":"172.21.15.136:6815/239937373","heartbeat_front_addr":"172.21.15.136:6814/239937373","state":["exists","up"]},{"osd":7,"uuid":"db593666-b9e5-4cee-b022-f243b58345cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6808","nonce":2884573928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6809","nonce":2884573928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6811","nonce":2884573928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6810","nonce":2884573928}]},"public_addr":"172.21.15.136:6808/2884573928","cluster_addr":"172.21.15.136:6809/2884573928","heartbeat_back_addr":"172.21.15.136:6811/2884573928","heartbeat_front_addr":"172.21.15.136:6810/2884573928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.320324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.405374+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.374617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.391393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.323092+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.319483+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.375307+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.301421+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:35:59.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.871+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 >> v1:172.21.15.110:6816/783516658 conn(0x7f738406c660 legacy=0x7f738406eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.871+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 >> v1:172.21.15.110:6790/0 conn(0x7f73a40726b0 legacy=0x7f73a4141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.871+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 shutdown_connections 2024-05-04T17:35:59.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.871+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 >> 172.21.15.110:0/657488189 conn(0x7f73a40b7b20 msgr2=0x7f73a406a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.871+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 shutdown_connections 2024-05-04T17:35:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.871+0000 7f73a8ba0640 1 -- 172.21.15.110:0/657488189 wait complete. 2024-05-04T17:35:59.884 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-04T17:35:59.885 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.980+0000 7f900b07d640 1 Processor -- start 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.980+0000 7f900b07d640 1 -- start start 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.980+0000 7f900b07d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90040afea0 con 0x7f90040ae6e0 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.980+0000 7f900b07d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9004072660 con 0x7f90040abe00 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.980+0000 7f900b07d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f900413c670 con 0x7f900413c260 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f900961a640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f900413c260 0x7f900414a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46286/0 (socket says 172.21.15.110:46286) 2024-05-04T17:35:59.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f900961a640 1 -- 172.21.15.110:0/1701652256 learned_addr learned my addr 172.21.15.110:0/1701652256 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f9008e19640 1 --1- 172.21.15.110:0/1701652256 >> v1:172.21.15.110:6789/0 conn(0x7f90040ae6e0 0x7f90040aad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35968/0 (socket says 172.21.15.110:35968) 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208503694 0 0) 0x7f90040afea0 con 0x7f90040ae6e0 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ff4003520 con 0x7f90040ae6e0 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807191335 0 0) 0x7f900413c670 con 0x7f900413c260 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90040afea0 con 0x7f900413c260 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506528131 0 0) 0x7f9004072660 con 0x7f90040abe00 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f900413c670 con 0x7f90040abe00 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 999798997 0 0) 0x7f90040afea0 con 0x7f900413c260 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 >> v1:172.21.15.136:6789/0 conn(0x7f90040abe00 legacy=0x7f90040ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 >> v1:172.21.15.110:6789/0 conn(0x7f90040ae6e0 legacy=0x7f90040aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900413c840 con 0x7f900413c260 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8ffc0018f0 con 0x7f900413c260 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ffc001c50 con 0x7f900413c260 2024-05-04T17:35:59.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.981+0000 7f90037fe640 1 -- 172.21.15.110:0/1701652256 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8ffc002060 con 0x7f900413c260 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.982+0000 7f900b07d640 1 -- 172.21.15.110:0/1701652256 >> v1:172.21.15.110:6790/0 conn(0x7f900413c260 legacy=0x7f900414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.982+0000 7f900b07d640 1 -- 172.21.15.110:0/1701652256 shutdown_connections 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.982+0000 7f900b07d640 1 -- 172.21.15.110:0/1701652256 >> 172.21.15.110:0/1701652256 conn(0x7f90040b3c50 msgr2=0x7f90040b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.982+0000 7f900b07d640 1 -- 172.21.15.110:0/1701652256 shutdown_connections 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.982+0000 7f900b07d640 1 -- 172.21.15.110:0/1701652256 wait complete. 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f900b07d640 1 Processor -- start 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f900b07d640 1 -- start start 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f900b07d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f900413daa0 con 0x7f900413c260 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f900b07d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9004071010 con 0x7f90040ae6e0 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f900b07d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90040711e0 con 0x7f90040abe00 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f9008e19640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f90040abe00 0x7f9004071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46298/0 (socket says 172.21.15.110:46298) 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.983+0000 7f9008e19640 1 -- 172.21.15.110:0/2917007072 learned_addr learned my addr 172.21.15.110:0/2917007072 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310220810 0 0) 0x7f90040711e0 con 0x7f90040abe00 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fe8003520 con 0x7f90040abe00 2024-05-04T17:35:59.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795865871 0 0) 0x7f900413daa0 con 0x7f900413c260 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90040711e0 con 0x7f900413c260 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799168058 0 0) 0x7f9004071010 con 0x7f90040ae6e0 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f900413daa0 con 0x7f90040ae6e0 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3493990436 0 0) 0x7f90040711e0 con 0x7f900413c260 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9004071010 con 0x7f900413c260 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4055355291 0 0) 0x7f8fe8003520 con 0x7f90040abe00 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90040711e0 con 0x7f90040abe00 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8ffc0018b0 con 0x7f900413c260 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8ff8001a30 con 0x7f90040abe00 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 491921546 0 0) 0x7f900413daa0 con 0x7f90040ae6e0 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fe8003520 con 0x7f90040ae6e0 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8ff0001870 con 0x7f90040ae6e0 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4197189357 0 0) 0x7f9004071010 con 0x7f900413c260 2024-05-04T17:35:59.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 >> v1:172.21.15.110:6790/0 conn(0x7f90040abe00 legacy=0x7f9004071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 >> v1:172.21.15.136:6789/0 conn(0x7f90040ae6e0 legacy=0x7f900406ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90041e1b10 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f90041deae0 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f90041defb0 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ffc001c50 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.984+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8ffc004fa0 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.985+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe4005080 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.986+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f8ffc01b110 con 0x7f900413c260 2024-05-04T17:35:59.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.986+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 4090+0+0 (unknown 1923690868 0 0) 0x7f8ffc086230 con 0x7f900413c260 2024-05-04T17:35:59.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:35:59.991+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f8ffc0551a0 con 0x7f900413c260 2024-05-04T17:36:00.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.168+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8fe4005370 con 0x7f900413c260 2024-05-04T17:36:00.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.169+0000 7f90017fa640 1 -- 172.21.15.110:0/2917007072 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12816 (unknown 1054498076 0 2811290940) 0x7f8ffc054aa0 con 0x7f900413c260 2024-05-04T17:36:00.170 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:36:00.170 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"fsid":"5cbb7704-6b22-42de-b82d-75ed642c3bca","created":"2024-05-04T17:35:47.067563+0000","modified":"2024-05-04T17:35:59.164344+0000","last_up_change":"2024-05-04T17:35:57.156954+0000","last_in_change":"2024-05-04T17:35:49.180095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:35:57.156872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-04T17:35:57.779672+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac107878-2a9c-435d-a24b-634939607eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6800","nonce":1712129713}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6801","nonce":1712129713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6803","nonce":1712129713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6802","nonce":1712129713}]},"public_addr":"172.21.15.110:6800/1712129713","cluster_addr":"172.21.15.110:6801/1712129713","heartbeat_back_addr":"172.21.15.110:6803/1712129713","heartbeat_front_addr":"172.21.15.110:6802/1712129713","state":["exists","up"]},{"osd":1,"uuid":"895a609b-83a1-4d9d-a323-9e777df46326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6808","nonce":1520708426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6810","nonce":1520708426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6814","nonce":1520708426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6812","nonce":1520708426}]},"public_addr":"172.21.15.110:6808/1520708426","cluster_addr":"172.21.15.110:6810/1520708426","heartbeat_back_addr":"172.21.15.110:6814/1520708426","heartbeat_front_addr":"172.21.15.110:6812/1520708426","state":["exists","up"]},{"osd":2,"uuid":"be5c569a-c157-48d0-b8d0-f44918cb1204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6804","nonce":4241686922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6805","nonce":4241686922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6807","nonce":4241686922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6806","nonce":4241686922}]},"public_addr":"172.21.15.110:6804/4241686922","cluster_addr":"172.21.15.110:6805/4241686922","heartbeat_back_addr":"172.21.15.110:6807/4241686922","heartbeat_front_addr":"172.21.15.110:6806/4241686922","state":["exists","up"]},{"osd":3,"uuid":"ff7aaeae-b017-4ece-8dab-2adea4056df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6809","nonce":2632237368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6811","nonce":2632237368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6815","nonce":2632237368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6813","nonce":2632237368}]},"public_addr":"172.21.15.110:6809/2632237368","cluster_addr":"172.21.15.110:6811/2632237368","heartbeat_back_addr":"172.21.15.110:6815/2632237368","heartbeat_front_addr":"172.21.15.110:6813/2632237368","state":["exists","up"]},{"osd":4,"uuid":"98a44aae-f478-4906-9f02-c63ae35f54d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6804","nonce":2025414972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6805","nonce":2025414972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6807","nonce":2025414972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6806","nonce":2025414972}]},"public_addr":"172.21.15.136:6804/2025414972","cluster_addr":"172.21.15.136:6805/2025414972","heartbeat_back_addr":"172.21.15.136:6807/2025414972","heartbeat_front_addr":"172.21.15.136:6806/2025414972","state":["exists","up"]},{"osd":5,"uuid":"c625ba28-d2ca-4ecb-8de2-4fecc934e626","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6800","nonce":2872378682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6801","nonce":2872378682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6803","nonce":2872378682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6802","nonce":2872378682}]},"public_addr":"172.21.15.136:6800/2872378682","cluster_addr":"172.21.15.136:6801/2872378682","heartbeat_back_addr":"172.21.15.136:6803/2872378682","heartbeat_front_addr":"172.21.15.136:6802/2872378682","state":["exists","up"]},{"osd":6,"uuid":"ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6812","nonce":239937373}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6813","nonce":239937373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6815","nonce":239937373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6814","nonce":239937373}]},"public_addr":"172.21.15.136:6812/239937373","cluster_addr":"172.21.15.136:6813/239937373","heartbeat_back_addr":"172.21.15.136:6815/239937373","heartbeat_front_addr":"172.21.15.136:6814/239937373","state":["exists","up"]},{"osd":7,"uuid":"db593666-b9e5-4cee-b022-f243b58345cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6808","nonce":2884573928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6809","nonce":2884573928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6811","nonce":2884573928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6810","nonce":2884573928}]},"public_addr":"172.21.15.136:6808/2884573928","cluster_addr":"172.21.15.136:6809/2884573928","heartbeat_back_addr":"172.21.15.136:6811/2884573928","heartbeat_front_addr":"172.21.15.136:6810/2884573928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.320324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.405374+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.374617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.391393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.323092+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.319483+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.375307+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.301421+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:36:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.171+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 >> v1:172.21.15.110:6816/783516658 conn(0x7f8fe806c9f0 legacy=0x7f8fe806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.171+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 >> v1:172.21.15.110:6789/0 conn(0x7f900413c260 legacy=0x7f900406f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.171+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 shutdown_connections 2024-05-04T17:36:00.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.171+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 >> 172.21.15.110:0/2917007072 conn(0x7f90040b3c50 msgr2=0x7f90040ad560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.171+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 shutdown_connections 2024-05-04T17:36:00.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.171+0000 7f900b07d640 1 -- 172.21.15.110:0/2917007072 wait complete. 2024-05-04T17:36:00.184 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-04T17:36:00.184 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-04T17:36:00.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-04T17:36:00.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-04T17:36:00.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-04T17:36:00.186 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-04T17:36:00.186 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-04T17:36:00.186 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-04T17:36:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.351+0000 7fcb5ffff640 1 Processor -- start 2024-05-04T17:36:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.351+0000 7fcb5ffff640 1 -- start start 2024-05-04T17:36:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.352+0000 7f0fb80d2640 1 Processor -- start 2024-05-04T17:36:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.352+0000 7fcb5ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5802a110 con 0x7fcb5802b360 2024-05-04T17:36:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.352+0000 7fcb5ffff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb580c75a0 con 0x7fcb5802a630 2024-05-04T17:36:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.352+0000 7fcb5f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcb5802b360 0x7fcb580320b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36010/0 (socket says 172.21.15.110:36010) 2024-05-04T17:36:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.352+0000 7fcb5f7fe640 1 -- 172.21.15.110:0/2088738347 learned_addr learned my addr 172.21.15.110:0/2088738347 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.352+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2088738347 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb58033100 con 0x7fcb58006400 2024-05-04T17:36:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7f0fb80d2640 1 -- start start 2024-05-04T17:36:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736286662 0 0) 0x7fcb5802a110 con 0x7fcb5802b360 2024-05-04T17:36:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb48005730 con 0x7fcb5802b360 2024-05-04T17:36:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 503706178 0 0) 0x7fcb48005730 con 0x7fcb5802b360 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 >> v1:172.21.15.110:6790/0 conn(0x7fcb58006400 legacy=0x7fcb580345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 >> v1:172.21.15.136:6789/0 conn(0x7fcb5802a630 legacy=0x7fcb580c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb580327c0 con 0x7fcb5802b360 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fcb6006a050 con 0x7fcb5802b360 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7f7270ed8640 1 Processor -- start 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.353+0000 7f7270ed8640 1 -- start start 2024-05-04T17:36:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb6004fb10 con 0x7fcb5802b360 2024-05-04T17:36:00.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7fcb5dffb640 1 -- 172.21.15.110:0/2088738347 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fcb60076240 con 0x7fcb5802b360 2024-05-04T17:36:00.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f0fb80d2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa802c820 con 0x7f0fa802b4b0 2024-05-04T17:36:00.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f0fb80d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa802af90 con 0x7f0fa8006c60 2024-05-04T17:36:00.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f0fb80d2640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa802ab90 con 0x7f0fa8006880 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f7270ed8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c0abf10 con 0x7f726c072be0 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f7270ed8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c0a9b70 con 0x7f726c0aa120 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f7270ed8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c06e0e0 con 0x7f726c13efc0 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f726c072be0 0x7f726c0ac760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36026/0 (socket says 172.21.15.110:36026) 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726affd640 1 -- 172.21.15.110:0/3749156665 learned_addr learned my addr 172.21.15.110:0/3749156665 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 743104116 0 0) 0x7f726c0abf10 con 0x7f726c072be0 2024-05-04T17:36:00.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7254003520 con 0x7f726c072be0 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f0fb566d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0fa8006c60 0x7f0fa80c6d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:36998/0 (socket says 172.21.15.110:36998) 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.354+0000 7f0fb566d640 1 -- 172.21.15.110:0/3270375521 learned_addr learned my addr 172.21.15.110:0/3270375521 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175402355 0 0) 0x7f0fa802ab90 con 0x7f0fa8006880 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f9c003500 con 0x7f0fa8006880 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 776082764 0 0) 0x7f7254003520 con 0x7f726c072be0 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 >> v1:172.21.15.110:6790/0 conn(0x7f726c13efc0 legacy=0x7f726c1499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 >> v1:172.21.15.136:6789/0 conn(0x7f726c0aa120 legacy=0x7f726c13e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:36:00.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726c06e2b0 con 0x7f726c072be0 2024-05-04T17:36:00.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670787809 0 0) 0x7f0fa802af90 con 0x7f0fa8006c60 2024-05-04T17:36:00.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa802ab90 con 0x7f0fa8006c60 2024-05-04T17:36:00.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2088738347 >> v1:172.21.15.110:6789/0 conn(0x7fcb5802b360 legacy=0x7fcb580320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7264001990 con 0x7f726c072be0 2024-05-04T17:36:00.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f726a7fc640 1 -- 172.21.15.110:0/3749156665 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7264001c60 con 0x7f726c072be0 2024-05-04T17:36:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2088738347 shutdown_connections 2024-05-04T17:36:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2088738347 >> 172.21.15.110:0/2088738347 conn(0x7fcb580058e0 msgr2=0x7fcb58005cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f147f23e640 1 Processor -- start 2024-05-04T17:36:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.355+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2225146114 0 0) 0x7f0fa802ab90 con 0x7f0fa8006c60 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 >> v1:172.21.15.110:6790/0 conn(0x7f0fa8006880 legacy=0x7f0fa80c0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 >> v1:172.21.15.110:6789/0 conn(0x7f0fa802b4b0 legacy=0x7f0fa80cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa80c7440 con 0x7f0fa8006c60 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0fb006a050 con 0x7f0fa8006c60 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fb004fb10 con 0x7f0fa8006c60 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f0fb4e6c640 1 -- 172.21.15.110:0/3270375521 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0fb0076240 con 0x7f0fa8006c60 2024-05-04T17:36:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2088738347 shutdown_connections 2024-05-04T17:36:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f0fb80d2640 1 -- 172.21.15.110:0/3270375521 >> v1:172.21.15.136:6789/0 conn(0x7f0fa8006c60 legacy=0x7f0fa80c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2088738347 wait complete. 2024-05-04T17:36:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7fd8b15ec640 1 Processor -- start 2024-05-04T17:36:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7fd8b15ec640 1 -- start start 2024-05-04T17:36:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f7270ed8640 1 -- 172.21.15.110:0/3749156665 >> v1:172.21.15.110:6789/0 conn(0x7f726c072be0 legacy=0x7f726c0ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.356+0000 7f147f23e640 1 -- start start 2024-05-04T17:36:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f0fb80d2640 1 -- 172.21.15.110:0/3270375521 shutdown_connections 2024-05-04T17:36:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f0fb80d2640 1 -- 172.21.15.110:0/3270375521 >> 172.21.15.110:0/3270375521 conn(0x7f0fa80058e0 msgr2=0x7f0fa8005cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f0fb80d2640 1 -- 172.21.15.110:0/3270375521 shutdown_connections 2024-05-04T17:36:00.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f147f23e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147002c810 con 0x7f1470034840 2024-05-04T17:36:00.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f147cfda640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1470034840 0x7f14700c6e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36050/0 (socket says 172.21.15.110:36050) 2024-05-04T17:36:00.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f147cfda640 1 -- 172.21.15.110:0/2771075893 learned_addr learned my addr 172.21.15.110:0/2771075893 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147002a110 con 0x7f14700327c0 2024-05-04T17:36:00.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14700c75a0 con 0x7f147002a630 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f0fb80d2640 1 -- 172.21.15.110:0/3270375521 wait complete. 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781468797 0 0) 0x7f147002c810 con 0x7f1470034840 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7fd8b15ec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac0aaeb0 con 0x7fd8ac141ff0 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7fd8b15ec640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac0ab120 con 0x7fd8ac14b900 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7fd8b15ec640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac0b3d20 con 0x7fd8ac0b4820 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7fd8b0deb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd8ac14b900 0x7fd8ac14dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37016/0 (socket says 172.21.15.110:37016) 2024-05-04T17:36:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.357+0000 7fd8b0deb640 1 -- 172.21.15.110:0/1379002441 learned_addr learned my addr 172.21.15.110:0/1379002441 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1460003520 con 0x7f1470034840 2024-05-04T17:36:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346772261 0 0) 0x7f147002a110 con 0x7f14700327c0 2024-05-04T17:36:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f147002c810 con 0x7f14700327c0 2024-05-04T17:36:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7fcb5ffff640 1 Processor -- start 2024-05-04T17:36:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7fcb5ffff640 1 -- start start 2024-05-04T17:36:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744759502 0 0) 0x7fd8ac0ab120 con 0x7fd8ac14b900 2024-05-04T17:36:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd894003520 con 0x7fd8ac14b900 2024-05-04T17:36:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f7270ed8640 1 -- 172.21.15.110:0/3749156665 shutdown_connections 2024-05-04T17:36:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f7270ed8640 1 -- 172.21.15.110:0/3749156665 >> 172.21.15.110:0/3749156665 conn(0x7f726c0b3830 msgr2=0x7f726c0b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f04635d4640 1 Processor -- start 2024-05-04T17:36:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3288197351 0 0) 0x7f147002c810 con 0x7f14700327c0 2024-05-04T17:36:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 >> v1:172.21.15.110:6790/0 conn(0x7f147002a630 legacy=0x7f14700320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:36:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 >> v1:172.21.15.110:6789/0 conn(0x7f1470034840 legacy=0x7f14700c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1470033100 con 0x7f14700327c0 2024-05-04T17:36:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f147806a050 con 0x7f14700327c0 2024-05-04T17:36:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f04635d4640 1 -- start start 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f0fb80d2640 1 Processor -- start 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f0fb80d2640 1 -- start start 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb5f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcb5802b360 0x7fcb580ba690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36070/0 (socket says 172.21.15.110:36070) 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb5f7fe640 1 -- 172.21.15.110:0/2597916595 learned_addr learned my addr 172.21.15.110:0/2597916595 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb580bada0 con 0x7fcb5802b360 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb580c47e0 con 0x7fcb5802a630 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5815cdb0 con 0x7fcb58006400 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502001991 0 0) 0x7fcb580bada0 con 0x7fcb5802b360 2024-05-04T17:36:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb4c003520 con 0x7fcb5802b360 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3966434165 0 0) 0x7fd894003520 con 0x7fd8ac14b900 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 >> v1:172.21.15.110:6790/0 conn(0x7fd8ac0b4820 legacy=0x7fd8ac13f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 >> v1:172.21.15.110:6789/0 conn(0x7fd8ac141ff0 legacy=0x7fd8ac142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac0a9990 con 0x7fd8ac14b900 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fd8a4001990 con 0x7fd8ac14b900 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f7270ed8640 1 -- 172.21.15.110:0/3749156665 shutdown_connections 2024-05-04T17:36:00.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.358+0000 7f7270ed8640 1 -- 172.21.15.110:0/3749156665 wait complete. 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7f04635d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0b4340 con 0x7f045c144020 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7f04635d4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0b3a90 con 0x7f045c0b1f60 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7f04635d4640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0b3de0 con 0x7f045c146d90 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f0461b71640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f045c146d90 0x7f045c151f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46390/0 (socket says 172.21.15.110:46390) 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f0461b71640 1 -- 172.21.15.110:0/2645378935 learned_addr learned my addr 172.21.15.110:0/2645378935 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fa3ea4b3640 1 Processor -- start 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7fa3ea4b3640 1 -- start start 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fa3ea4b3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e4071400 con 0x7fa3e4071b00 2024-05-04T17:36:00.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fa3ea4b3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e4141870 con 0x7fa3e406f140 2024-05-04T17:36:00.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fa3ea4b3640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e4068a40 con 0x7fa3e41424a0 2024-05-04T17:36:00.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f147804fb10 con 0x7f14700327c0 2024-05-04T17:36:00.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7f14777fe640 1 -- 172.21.15.110:0/2771075893 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1478076240 con 0x7f14700327c0 2024-05-04T17:36:00.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 >> v1:172.21.15.136:6789/0 conn(0x7f14700327c0 legacy=0x7f14700cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3580883669 0 0) 0x7fcb4c003520 con 0x7fcb5802b360 2024-05-04T17:36:00.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb580bada0 con 0x7fcb5802b360 2024-05-04T17:36:00.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.359+0000 7f7270ed8640 1 Processor -- start 2024-05-04T17:36:00.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f7270ed8640 1 -- start start 2024-05-04T17:36:00.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fcb60068e20 con 0x7fcb5802b360 2024-05-04T17:36:00.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8a4001ba0 con 0x7fd8ac14b900 2024-05-04T17:36:00.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fd8aaffd640 1 -- 172.21.15.110:0/1379002441 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fd8a4001ff0 con 0x7fd8ac14b900 2024-05-04T17:36:00.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538405923 0 0) 0x7fcb580c47e0 con 0x7fcb5802a630 2024-05-04T17:36:00.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fa3e3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa3e406f140 0x7fa3e4141d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37042/0 (socket says 172.21.15.110:37042) 2024-05-04T17:36:00.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fa3e3fff640 1 -- 172.21.15.110:0/2925459344 learned_addr learned my addr 172.21.15.110:0/2925459344 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097297968 0 0) 0x7f045c0b4340 con 0x7f045c144020 2024-05-04T17:36:00.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0448003520 con 0x7f045c144020 2024-05-04T17:36:00.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 shutdown_connections 2024-05-04T17:36:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb4c003520 con 0x7fcb5802a630 2024-05-04T17:36:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.360+0000 7fd8b15ec640 1 -- 172.21.15.110:0/1379002441 >> v1:172.21.15.136:6789/0 conn(0x7fd8ac14b900 legacy=0x7fd8ac14dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974900496 0 0) 0x7f045c0b3a90 con 0x7f045c0b1f60 2024-05-04T17:36:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 >> 172.21.15.110:0/2771075893 conn(0x7f14700058e0 msgr2=0x7f1470005cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fd8b15ec640 1 -- 172.21.15.110:0/1379002441 shutdown_connections 2024-05-04T17:36:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fd8b15ec640 1 -- 172.21.15.110:0/1379002441 >> 172.21.15.110:0/1379002441 conn(0x7fd8ac061c70 msgr2=0x7fd8ac062080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fd8b15ec640 1 -- 172.21.15.110:0/1379002441 shutdown_connections 2024-05-04T17:36:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0448003880 con 0x7f045c0b1f60 2024-05-04T17:36:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f7270ed8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c11c700 con 0x7f726c13efc0 2024-05-04T17:36:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f7270ed8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c11c9e0 con 0x7f726c11c1f0 2024-05-04T17:36:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fd8b15ec640 1 -- 172.21.15.110:0/1379002441 wait complete. 2024-05-04T17:36:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f7270ed8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c11ccc0 con 0x7f726c0aa120 2024-05-04T17:36:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981742884 0 0) 0x7fcb5815cdb0 con 0x7fcb58006400 2024-05-04T17:36:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655794165 0 0) 0x7fa3e4141870 con 0x7fa3e406f140 2024-05-04T17:36:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3cc003520 con 0x7fa3e406f140 2024-05-04T17:36:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f726affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f726c13efc0 0x7f726c1d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36086/0 (socket says 172.21.15.110:36086) 2024-05-04T17:36:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f726affd640 1 -- 172.21.15.110:0/3853788194 learned_addr learned my addr 172.21.15.110:0/3853788194 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493759556 0 0) 0x7fa3e4068a40 con 0x7fa3e41424a0 2024-05-04T17:36:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3e4141870 con 0x7fa3e41424a0 2024-05-04T17:36:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 shutdown_connections 2024-05-04T17:36:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fe5b2722640 1 Processor -- start 2024-05-04T17:36:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f147f23e640 1 -- 172.21.15.110:0/2771075893 wait complete. 2024-05-04T17:36:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb580c47e0 con 0x7fcb58006400 2024-05-04T17:36:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3285332695 0 0) 0x7fa3cc003520 con 0x7fa3e406f140 2024-05-04T17:36:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f0fb80d2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa80c3d20 con 0x7f0fa8006880 2024-05-04T17:36:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 >> v1:172.21.15.110:6790/0 conn(0x7fa3e41424a0 legacy=0x7fa3e413b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f0fb80d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa8031450 con 0x7f0fa80c4420 2024-05-04T17:36:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f0fb80d2640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fa80c2c10 con 0x7f0fa802b4b0 2024-05-04T17:36:00.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fe5b2722640 1 -- start start 2024-05-04T17:36:00.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f0fb566d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0fa802b4b0 0x7f0fa8030d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46402/0 (socket says 172.21.15.110:46402) 2024-05-04T17:36:00.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f0fb566d640 1 -- 172.21.15.110:0/1953941301 learned_addr learned my addr 172.21.15.110:0/1953941301 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 >> v1:172.21.15.110:6789/0 conn(0x7fa3e4071b00 legacy=0x7fa3e4149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e4068c10 con 0x7fa3e406f140 2024-05-04T17:36:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7f0fb666f640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0fa80c4420 0x7f0fa80c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37060/0 (socket says 172.21.15.110:37060) 2024-05-04T17:36:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa3d4001610 con 0x7fa3e406f140 2024-05-04T17:36:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.361+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2472210621 0 0) 0x7fcb4c003520 con 0x7fcb5802a630 2024-05-04T17:36:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb5815cdb0 con 0x7fcb5802a630 2024-05-04T17:36:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3510438840 0 0) 0x7fcb580c47e0 con 0x7fcb58006400 2024-05-04T17:36:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb4c003520 con 0x7fcb58006400 2024-05-04T17:36:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3d4001870 con 0x7fa3e406f140 2024-05-04T17:36:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fa3e2ffd640 1 -- 172.21.15.110:0/2925459344 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa3d4001c40 con 0x7fa3e406f140 2024-05-04T17:36:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fcb54001830 con 0x7fcb5802a630 2024-05-04T17:36:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fcb500016c0 con 0x7fcb58006400 2024-05-04T17:36:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 500723303 0 0) 0x7f0fa80c2c10 con 0x7f0fa802b4b0 2024-05-04T17:36:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f94003520 con 0x7f0fa802b4b0 2024-05-04T17:36:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1609992808 0 0) 0x7fcb5815cdb0 con 0x7fcb5802a630 2024-05-04T17:36:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 >> v1:172.21.15.110:6790/0 conn(0x7fcb58006400 legacy=0x7fcb580c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 >> v1:172.21.15.110:6789/0 conn(0x7fcb5802b360 legacy=0x7fcb580ba690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3517051496 0 0) 0x7f045c0b3de0 con 0x7f045c146d90 2024-05-04T17:36:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5815cf80 con 0x7fcb5802a630 2024-05-04T17:36:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f045c0b3a90 con 0x7f045c146d90 2024-05-04T17:36:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3332759826 0 0) 0x7f0448003880 con 0x7f045c0b1f60 2024-05-04T17:36:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 >> v1:172.21.15.110:6790/0 conn(0x7f045c146d90 legacy=0x7f045c151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fe5b2722640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a4008f20 con 0x7fe5a4006e10 2024-05-04T17:36:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fe5b2722640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a402bea0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fe5b2722640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a402c180 con 0x7fe5a4008570 2024-05-04T17:36:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 >> v1:172.21.15.110:6789/0 conn(0x7f045c144020 legacy=0x7f045c146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fd8b15ec640 1 Processor -- start 2024-05-04T17:36:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c06d5e0 con 0x7f045c0b1f60 2024-05-04T17:36:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04540018f0 con 0x7f045c0b1f60 2024-05-04T17:36:00.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/2925459344 >> v1:172.21.15.136:6789/0 conn(0x7fa3e406f140 legacy=0x7fa3e4141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696403550 0 0) 0x7f0fa8031450 con 0x7f0fa80c4420 2024-05-04T17:36:00.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fe5b1f21640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe5a40cf8a0 0x7fe5a40d7530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37074/0 (socket says 172.21.15.110:37074) 2024-05-04T17:36:00.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fe5b1f21640 1 -- 172.21.15.110:0/2549739382 learned_addr learned my addr 172.21.15.110:0/2549739382 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb540035f0 con 0x7fcb5802a630 2024-05-04T17:36:00.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/2925459344 shutdown_connections 2024-05-04T17:36:00.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.362+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/2925459344 >> 172.21.15.110:0/2925459344 conn(0x7fa3e40b4210 msgr2=0x7fa3e40b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa80c2c10 con 0x7f0fa80c4420 2024-05-04T17:36:00.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fcb54003940 con 0x7fcb5802a630 2024-05-04T17:36:00.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/2925459344 shutdown_connections 2024-05-04T17:36:00.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/2925459344 wait complete. 2024-05-04T17:36:00.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230929953 0 0) 0x7fe5a402bea0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe598003520 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb580bafc0 con 0x7fcb5802a630 2024-05-04T17:36:00.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb58031200 con 0x7fcb5802a630 2024-05-04T17:36:00.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968575568 0 0) 0x7fe5a402c180 con 0x7fe5a4008570 2024-05-04T17:36:00.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a402bea0 con 0x7fe5a4008570 2024-05-04T17:36:00.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f147f23e640 1 Processor -- start 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1323419438 0 0) 0x7f0f94003520 con 0x7f0fa802b4b0 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fa8031450 con 0x7f0fa802b4b0 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0fb0068e20 con 0x7f0fa802b4b0 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1319604148 0 0) 0x7f0fa80c2c10 con 0x7f0fa80c4420 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f94003520 con 0x7f0fa80c4420 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0fac001810 con 0x7f0fa80c4420 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 937264529 0 0) 0x7f0f94003520 con 0x7f0fa80c4420 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 >> v1:172.21.15.110:6790/0 conn(0x7f0fa802b4b0 legacy=0x7f0fa8030d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 >> v1:172.21.15.110:6789/0 conn(0x7f0fa8006880 legacy=0x7f0fa8158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa80c2de0 con 0x7f0fa80c4420 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1244204238 0 0) 0x7f726c11c9e0 con 0x7f726c11c1f0 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7258003520 con 0x7f726c11c1f0 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461275346 0 0) 0x7f726c11c700 con 0x7f726c13efc0 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f726c11c9e0 con 0x7f726c13efc0 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1581173551 0 0) 0x7fe598003520 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 >> v1:172.21.15.110:6790/0 conn(0x7fe5a4008570 legacy=0x7fe5a40cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 >> v1:172.21.15.110:6789/0 conn(0x7fe5a4006e10 legacy=0x7fe5a40c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a402c460 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe5ac0504e0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.363+0000 7fd8b15ec640 1 -- start start 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8b15ec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac1341c0 con 0x7fd8ac0b4820 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8b15ec640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac133940 con 0x7fd8ac14b900 2024-05-04T17:36:00.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8b15ec640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac0b5920 con 0x7fd8ac141ff0 2024-05-04T17:36:00.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 115385988 0 0) 0x7f7258003520 con 0x7f726c11c1f0 2024-05-04T17:36:00.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f726c11c700 con 0x7f726c11c1f0 2024-05-04T17:36:00.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7260001600 con 0x7f726c11c1f0 2024-05-04T17:36:00.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5ac04fb10 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fe5a3fff640 1 -- 172.21.15.110:0/2549739382 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe5ac0008d0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8b0deb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd8ac14b900 0x7fd8ac134a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37084/0 (socket says 172.21.15.110:37084) 2024-05-04T17:36:00.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8b0deb640 1 -- 172.21.15.110:0/572030017 learned_addr learned my addr 172.21.15.110:0/572030017 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fa3ea4b3640 1 Processor -- start 2024-05-04T17:36:00.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fac0035f0 con 0x7f0fa80c4420 2024-05-04T17:36:00.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0fac001670 con 0x7f0fa80c4420 2024-05-04T17:36:00.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1994270544 0 0) 0x7f726c11c9e0 con 0x7f726c13efc0 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7258003520 con 0x7f726c13efc0 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7264004ef0 con 0x7f726c13efc0 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791417766 0 0) 0x7fd8ac133940 con 0x7fd8ac14b900 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd898003520 con 0x7fd8ac14b900 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2965076621 0 0) 0x7fd8ac0b5920 con 0x7fd8ac141ff0 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8ac133940 con 0x7fd8ac141ff0 2024-05-04T17:36:00.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f147f23e640 1 -- start start 2024-05-04T17:36:00.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fcb457fa640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.136:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fcb38000f80 con 0x7fcb5802a630 2024-05-04T17:36:00.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1048795749 0 0) 0x7fd8ac133940 con 0x7fd8ac141ff0 2024-05-04T17:36:00.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8ac0b5920 con 0x7fd8ac141ff0 2024-05-04T17:36:00.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fd8a0001760 con 0x7fd8ac141ff0 2024-05-04T17:36:00.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fe5b2722640 1 -- 172.21.15.110:0/2549739382 >> v1:172.21.15.136:6789/0 conn(0x7fe5a40cf8a0 legacy=0x7fe5a40d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fe5b2722640 1 -- 172.21.15.110:0/2549739382 shutdown_connections 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fe5b2722640 1 -- 172.21.15.110:0/2549739382 >> 172.21.15.110:0/2549739382 conn(0x7fe5a4005c60 msgr2=0x7fe5a4006070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7fa3ea4b3640 1 -- start start 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fa3ea4b3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e40acd60 con 0x7fa3e406f140 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fa3ea4b3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e412b8e0 con 0x7fa3e41424a0 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fa3ea4b3640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e413e140 con 0x7fa3e4071b00 2024-05-04T17:36:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fe5b2722640 1 -- 172.21.15.110:0/2549739382 shutdown_connections 2024-05-04T17:36:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fa3e37fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fa3e4071b00 0x7fa3e412aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46422/0 (socket says 172.21.15.110:46422) 2024-05-04T17:36:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fa3e8a50640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa3e41424a0 0x7fa3e412b1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37094/0 (socket says 172.21.15.110:37094) 2024-05-04T17:36:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fa3e37fe640 1 -- 172.21.15.110:0/871353311 learned_addr learned my addr 172.21.15.110:0/871353311 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 24951550 0 0) 0x7f7258003520 con 0x7f726c13efc0 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 >> v1:172.21.15.110:6790/0 conn(0x7f726c0aa120 legacy=0x7f726c11b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 >> v1:172.21.15.136:6789/0 conn(0x7f726c11c1f0 legacy=0x7f726c1dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726c071f30 con 0x7f726c13efc0 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7264005340 con 0x7f726c13efc0 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7264005710 con 0x7f726c13efc0 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7fe5b2722640 1 -- 172.21.15.110:0/2549739382 wait complete. 2024-05-04T17:36:00.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fcb54019ab0 con 0x7fcb5802a630 2024-05-04T17:36:00.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762298153 0 0) 0x7fa3e413e140 con 0x7fa3e4071b00 2024-05-04T17:36:00.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0454001b50 con 0x7f045c0b1f60 2024-05-04T17:36:00.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d0003520 con 0x7fa3e4071b00 2024-05-04T17:36:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.364+0000 7f044ffff640 1 -- 172.21.15.110:0/2645378935 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0454001fa0 con 0x7f045c0b1f60 2024-05-04T17:36:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f04635d4640 1 -- 172.21.15.110:0/2645378935 >> v1:172.21.15.136:6789/0 conn(0x7f045c0b1f60 legacy=0x7f045c143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f04635d4640 1 -- 172.21.15.110:0/2645378935 shutdown_connections 2024-05-04T17:36:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f04635d4640 1 -- 172.21.15.110:0/2645378935 >> 172.21.15.110:0/2645378935 conn(0x7f045c061c70 msgr2=0x7f045c062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f04635d4640 1 -- 172.21.15.110:0/2645378935 shutdown_connections 2024-05-04T17:36:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f04635d4640 1 -- 172.21.15.110:0/2645378935 wait complete. 2024-05-04T17:36:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f147cfda640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f147002a630 0x7f14700c3100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37104/0 (socket says 172.21.15.110:37104) 2024-05-04T17:36:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f1477fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f14700327c0 0x7f14700bad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36134/0 (socket says 172.21.15.110:36134) 2024-05-04T17:36:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f147d7db640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f1470034840 0x7f1470158410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46438/0 (socket says 172.21.15.110:46438) 2024-05-04T17:36:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f147cfda640 1 -- 172.21.15.110:0/2960692571 learned_addr learned my addr 172.21.15.110:0/2960692571 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147002cf10 con 0x7f14700327c0 2024-05-04T17:36:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.365+0000 7f0fb80d2640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa8031940 con 0x7f0fa80c4420 2024-05-04T17:36:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f0fb80d2640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0fa8165070 con 0x7f0fa80c4420 2024-05-04T17:36:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14700c3b10 con 0x7f147002a630 2024-05-04T17:36:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fcb54084cf0 con 0x7fcb5802a630 2024-05-04T17:36:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001151191 0 0) 0x7f147002cf10 con 0x7f14700327c0 2024-05-04T17:36:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6808/1520708426 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb4c0776c0 con 0x7fcb4c073af0 2024-05-04T17:36:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7fcb540850d0 con 0x7fcb5802a630 2024-05-04T17:36:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f726c072240 con 0x7f726c13efc0 2024-05-04T17:36:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f726c1dee50 con 0x7f726c13efc0 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.136:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0f8c000f80 con 0x7f0fa80c4420 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1401949756 0 0) 0x7fa3e412b8e0 con 0x7fa3e41424a0 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3e413e140 con 0x7fa3e41424a0 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3524040846 0 0) 0x7fa3d0003520 con 0x7fa3e4071b00 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3e412b8e0 con 0x7fa3e4071b00 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837238601 0 0) 0x7fa3e40acd60 con 0x7fa3e406f140 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d0003520 con 0x7fa3e406f140 2024-05-04T17:36:00.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa3dc001bb0 con 0x7fa3e4071b00 2024-05-04T17:36:00.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14700c3810 con 0x7f1470034840 2024-05-04T17:36:00.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1464003520 con 0x7f14700327c0 2024-05-04T17:36:00.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f04635d4640 1 Processor -- start 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135834412 0 0) 0x7f14700c3b10 con 0x7f147002a630 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f04635d4640 1 -- start start 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3809125877 0 0) 0x7fd898003520 con 0x7fd8ac14b900 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8ac133940 con 0x7fd8ac14b900 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896894443 0 0) 0x7fd8ac1341c0 con 0x7fd8ac0b4820 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd898003520 con 0x7fd8ac0b4820 2024-05-04T17:36:00.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 204917478 0 0) 0x7fd8ac0b5920 con 0x7fd8ac141ff0 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 >> v1:172.21.15.136:6789/0 conn(0x7fd8ac14b900 legacy=0x7fd8ac134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 >> v1:172.21.15.110:6789/0 conn(0x7fd8ac0b4820 legacy=0x7fd8ac145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.366+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac146240 con 0x7fd8ac141ff0 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fd8b15ec640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ac1dc820 con 0x7fd8ac141ff0 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fd8b15ec640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8ac135140 con 0x7fd8ac141ff0 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f726c04fb10 con 0x7f726c13efc0 2024-05-04T17:36:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f0fac0039b0 con 0x7f0fa80c4420 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fe5b2722640 1 Processor -- start 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.367+0000 7fe5b2722640 1 -- start start 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fe5b2722640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a40cc5a0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fe5b2722640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a40cc880 con 0x7fe5a40cdfe0 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f0460b6f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f045c144020 0x7f045c14a120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36142/0 (socket says 172.21.15.110:36142) 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f0460b6f640 1 -- 172.21.15.110:0/2618623543 learned_addr learned my addr 172.21.15.110:0/2618623543 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c1dfe80 con 0x7f045c144020 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c1488a0 con 0x7f045c146d90 2024-05-04T17:36:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c1dfb30 con 0x7f045c0b1f60 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2621549748 0 0) 0x7fa3e412b8e0 con 0x7fa3e4071b00 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 >> v1:172.21.15.136:6789/0 conn(0x7fa3e41424a0 legacy=0x7fa3e412b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 >> v1:172.21.15.110:6789/0 conn(0x7fa3e406f140 legacy=0x7fa3e413e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e413e420 con 0x7fa3e4071b00 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e413c480 con 0x7fa3e4071b00 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3e4141570 con 0x7fa3e4071b00 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.110:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa3e404fe40 con 0x7fa3e4071b00 2024-05-04T17:36:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f147002cf10 con 0x7f147002a630 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2313885650 0 0) 0x7f14700c3810 con 0x7f1470034840 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.368+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14700c3b10 con 0x7f1470034840 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1677782885 0 0) 0x7f1464003520 con 0x7f14700327c0 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14700c3810 con 0x7f14700327c0 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f146c001860 con 0x7f14700327c0 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7fe5b0f1f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe5a40cf8a0 0x7fe5a40cb640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36154/0 (socket says 172.21.15.110:36154) 2024-05-04T17:36:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7fe5b0f1f640 1 -- 172.21.15.110:0/3636595386 learned_addr learned my addr 172.21.15.110:0/3636595386 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== osd.1 v1:172.21.15.110:6808/1520708426 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcb4c0776c0 con 0x7fcb4c073af0 2024-05-04T17:36:00.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931653161 0 0) 0x7f045c1dfe80 con 0x7f045c144020 2024-05-04T17:36:00.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0440003520 con 0x7f045c144020 2024-05-04T17:36:00.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28859426 0 0) 0x7f045c1488a0 con 0x7f045c146d90 2024-05-04T17:36:00.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f045c1dfe80 con 0x7f045c146d90 2024-05-04T17:36:00.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3418121927 0 0) 0x7f14700c3810 con 0x7f14700327c0 2024-05-04T17:36:00.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 >> v1:172.21.15.110:6790/0 conn(0x7f1470034840 legacy=0x7f1470158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 >> v1:172.21.15.136:6789/0 conn(0x7f147002a630 legacy=0x7f14700c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14700bf880 con 0x7f14700327c0 2024-05-04T17:36:00.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583763339 0 0) 0x7fe5a40cc5a0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8a0003610 con 0x7fd8ac141ff0 2024-05-04T17:36:00.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fd8a0003960 con 0x7fd8ac141ff0 2024-05-04T17:36:00.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f146c003610 con 0x7f14700327c0 2024-05-04T17:36:00.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f146c0039e0 con 0x7f14700327c0 2024-05-04T17:36:00.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a40c1130 con 0x7fe5a4006e10 2024-05-04T17:36:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14700bfb70 con 0x7f14700327c0 2024-05-04T17:36:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe590003520 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14700bb660 con 0x7f14700327c0 2024-05-04T17:36:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f0fac0525f0 con 0x7f0fa80c4420 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.110:6800/1712129713 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f94078670 con 0x7f0f94074a00 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f0fac050c90 con 0x7f0fa80c4420 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.369+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f7264001e90 con 0x7f726c13efc0 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f7264086700 con 0x7f726c13efc0 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6809/2632237368 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7258077750 con 0x7f7258073b80 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f7264086b00 con 0x7f726c13efc0 2024-05-04T17:36:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3539623837 0 0) 0x7fe590003520 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5a40cc5a0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.370+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe5a8001960 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2734169801 0 0) 0x7fe5a40cc5a0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3dc003990 con 0x7fa3e4071b00 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 >> v1:172.21.15.110:6790/0 conn(0x7fe5a4006e10 legacy=0x7fe5a40caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 >> v1:172.21.15.136:6789/0 conn(0x7fe5a40cdfe0 legacy=0x7fe5a40ba340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a40cbd50 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa3dc003ce0 con 0x7fa3e4071b00 2024-05-04T17:36:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a40baa50 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fa3dc003ec0 con 0x7fa3e4071b00 2024-05-04T17:36:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5a40bd290 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1470004cc0 con 0x7f14700327c0 2024-05-04T17:36:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== osd.0 v1:172.21.15.110:6800/1712129713 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0f94078670 con 0x7f0f94074a00 2024-05-04T17:36:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== osd.3 v1:172.21.15.110:6809/2632237368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7258077750 con 0x7f7258073b80 2024-05-04T17:36:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1301809836 0 0) 0x7f045c1dfe80 con 0x7f045c146d90 2024-05-04T17:36:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f045c1488a0 con 0x7f045c146d90 2024-05-04T17:36:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2800898390 0 0) 0x7f0440003520 con 0x7f045c144020 2024-05-04T17:36:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f045c1dfe80 con 0x7f045c144020 2024-05-04T17:36:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04580017b0 con 0x7f045c146d90 2024-05-04T17:36:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04500012b0 con 0x7f045c144020 2024-05-04T17:36:00.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5a80038a0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe5a8003c70 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.371+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd8ac04fb10 con 0x7fd8ac141ff0 2024-05-04T17:36:00.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.373+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fa3dc086550 con 0x7fa3e4071b00 2024-05-04T17:36:00.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3143303174 0 0) 0x7f045c1dfe80 con 0x7f045c144020 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 >> v1:172.21.15.110:6790/0 conn(0x7f045c0b1f60 legacy=0x7f045c1df420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 >> v1:172.21.15.136:6789/0 conn(0x7f045c146d90 legacy=0x7f045c1dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c14a830 con 0x7f045c144020 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.372+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f045c14aac0 con 0x7f045c144020 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.373+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f045c145ea0 con 0x7f045c144020 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.373+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.136:6812/239937373 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3d00777a0 con 0x7fa3d0073bd0 2024-05-04T17:36:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.373+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7fa3dc086900 con 0x7fa3e4071b00 2024-05-04T17:36:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.373+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f146c003b80 con 0x7f14700327c0 2024-05-04T17:36:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe58c000f80 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f146c086120 con 0x7f14700327c0 2024-05-04T17:36:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fd8a0019c70 con 0x7fd8ac141ff0 2024-05-04T17:36:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.136:6808/2884573928 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f146407bbb0 con 0x7f1464077fe0 2024-05-04T17:36:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f146c050ac0 con 0x7f14700327c0 2024-05-04T17:36:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04500035f0 con 0x7f045c144020 2024-05-04T17:36:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.374+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f04500039c0 con 0x7f045c144020 2024-05-04T17:36:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.375+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fe5a8003e90 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.375+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fd8a0084f30 con 0x7fd8ac141ff0 2024-05-04T17:36:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.375+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6804/4241686922 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd898077750 con 0x7fd898073ae0 2024-05-04T17:36:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.375+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7fd8a0085330 con 0x7fd8ac141ff0 2024-05-04T17:36:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.375+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f043c000f80 con 0x7f045c144020 2024-05-04T17:36:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f0450003c20 con 0x7f045c144020 2024-05-04T17:36:00.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fe5a80863e0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.136:6804/2025414972 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe590077630 con 0x7fe5900739c0 2024-05-04T17:36:00.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7fe5a80867e0 con 0x7fe5a40cf8a0 2024-05-04T17:36:00.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f0450086260 con 0x7f045c144020 2024-05-04T17:36:00.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.136:6800/2872378682 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0440077590 con 0x7f04400739c0 2024-05-04T17:36:00.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.376+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f0450086560 con 0x7f045c144020 2024-05-04T17:36:00.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.377+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== osd.7 v1:172.21.15.136:6808/2884573928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f146407bbb0 con 0x7f1464077fe0 2024-05-04T17:36:00.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.377+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== osd.2 v1:172.21.15.110:6804/4241686922 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd898077750 con 0x7fd898073ae0 2024-05-04T17:36:00.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.378+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== osd.6 v1:172.21.15.136:6812/239937373 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa3d00777a0 con 0x7fa3d0073bd0 2024-05-04T17:36:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.378+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== osd.4 v1:172.21.15.136:6804/2025414972 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe590077630 con 0x7fe5900739c0 2024-05-04T17:36:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.378+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== osd.5 v1:172.21.15.136:6800/2872378682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0440077590 con 0x7f04400739c0 2024-05-04T17:36:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.402+0000 7fcb457fa640 1 -- 172.21.15.110:0/2597916595 --> v1:172.21.15.110:6808/1520708426 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb38002c50 con 0x7fcb4c073af0 2024-05-04T17:36:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.403+0000 7fcb477fe640 1 -- 172.21.15.110:0/2597916595 <== osd.1 v1:172.21.15.110:6808/1520708426 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3371909001) 0x7fcb38002c50 con 0x7fcb4c073af0 2024-05-04T17:36:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.403+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 --> v1:172.21.15.110:6809/2632237368 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f726c11b2f0 con 0x7f7258073b80 2024-05-04T17:36:00.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.403+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 >> v1:172.21.15.110:6808/1520708426 conn(0x7fcb4c073af0 legacy=0x7fcb4c075f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.403+0000 7f7253fff640 1 -- 172.21.15.110:0/3853788194 <== osd.3 v1:172.21.15.110:6809/2632237368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 982563978) 0x7f726c11b2f0 con 0x7f7258073b80 2024-05-04T17:36:00.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.403+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 >> v1:172.21.15.110:6816/783516658 conn(0x7fcb4c06c9d0 legacy=0x7fcb4c06ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 >> v1:172.21.15.136:6789/0 conn(0x7fcb5802a630 legacy=0x7fcb580b9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 >> v1:172.21.15.110:6809/2632237368 conn(0x7f7258073b80 legacy=0x7f7258075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7fcb5f7fe640 1 -- 172.21.15.110:0/2597916595 reap_dead start 2024-05-04T17:36:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 shutdown_connections 2024-05-04T17:36:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 >> 172.21.15.110:0/2597916595 conn(0x7fcb580058e0 msgr2=0x7fcb580043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 >> v1:172.21.15.110:6816/783516658 conn(0x7f725806ca60 legacy=0x7f725806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 >> v1:172.21.15.110:6789/0 conn(0x7f726c13efc0 legacy=0x7f726c1d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7f726bfff640 1 -- 172.21.15.110:0/3853788194 reap_dead start 2024-05-04T17:36:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 shutdown_connections 2024-05-04T17:36:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.404+0000 7fcb5ffff640 1 -- 172.21.15.110:0/2597916595 wait complete. 2024-05-04T17:36:00.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.405+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 shutdown_connections 2024-05-04T17:36:00.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.405+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 >> 172.21.15.110:0/3853788194 conn(0x7f726c0b3830 msgr2=0x7f726c06a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.405+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 shutdown_connections 2024-05-04T17:36:00.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.405+0000 7f7270ed8640 1 -- 172.21.15.110:0/3853788194 wait complete. 2024-05-04T17:36:00.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.406+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 --> v1:172.21.15.110:6800/1712129713 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f8c002c50 con 0x7f0f94074a00 2024-05-04T17:36:00.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.407+0000 7f0fa67fc640 1 -- 172.21.15.110:0/1953941301 <== osd.0 v1:172.21.15.110:6800/1712129713 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 982563978) 0x7f0f8c002c50 con 0x7f0f94074a00 2024-05-04T17:36:00.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.408+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 >> v1:172.21.15.110:6800/1712129713 conn(0x7f0f94074a00 legacy=0x7f0f94076ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.408+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 >> v1:172.21.15.110:6816/783516658 conn(0x7f0f9406cfb0 legacy=0x7f0f9406f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.408+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 >> v1:172.21.15.136:6789/0 conn(0x7f0fa80c4420 legacy=0x7f0fa80c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.408+0000 7f0fb666f640 1 -- 172.21.15.110:0/1953941301 reap_dead start 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.409+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 shutdown_connections 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.409+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 >> 172.21.15.110:0/1953941301 conn(0x7f0fa80058e0 msgr2=0x7f0fa8038120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.409+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 shutdown_connections 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.409+0000 7f0f93fff640 1 -- 172.21.15.110:0/1953941301 wait complete. 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.409+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 --> v1:172.21.15.136:6808/2884573928 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1470158d70 con 0x7f1464077fe0 2024-05-04T17:36:00.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.410+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 --> v1:172.21.15.110:6804/4241686922 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd8ac0b5250 con 0x7fd898073ae0 2024-05-04T17:36:00.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.410+0000 7f14757fa640 1 -- 172.21.15.110:0/2960692571 <== osd.7 v1:172.21.15.136:6808/2884573928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1046766345) 0x7f1470158d70 con 0x7f1464077fe0 2024-05-04T17:36:00.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.410+0000 7fd8a8ff9640 1 -- 172.21.15.110:0/572030017 <== osd.2 v1:172.21.15.110:6804/4241686922 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3371909001) 0x7fd8ac0b5250 con 0x7fd898073ae0 2024-05-04T17:36:00.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.410+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 >> v1:172.21.15.136:6808/2884573928 conn(0x7f1464077fe0 legacy=0x7f146407a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 >> v1:172.21.15.110:6804/4241686922 conn(0x7fd898073ae0 legacy=0x7fd898075f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 >> v1:172.21.15.110:6816/783516658 conn(0x7fd89806ca50 legacy=0x7fd89806ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 >> v1:172.21.15.110:6790/0 conn(0x7fd8ac141ff0 legacy=0x7fd8ac1dc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 --> v1:172.21.15.136:6804/2025414972 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe58c002c50 con 0x7fe5900739c0 2024-05-04T17:36:00.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 --> v1:172.21.15.136:6800/2872378682 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f043c002c50 con 0x7f04400739c0 2024-05-04T17:36:00.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 >> v1:172.21.15.110:6816/783516658 conn(0x7f1464070ec0 legacy=0x7f1464073380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 >> v1:172.21.15.110:6789/0 conn(0x7f14700327c0 legacy=0x7f14700bad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8b0deb640 1 -- 172.21.15.110:0/572030017 reap_dead start 2024-05-04T17:36:00.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 shutdown_connections 2024-05-04T17:36:00.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 >> 172.21.15.110:0/572030017 conn(0x7fd8ac061c70 msgr2=0x7fd8ac13a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fe5a1ffb640 1 -- 172.21.15.110:0/3636595386 <== osd.4 v1:172.21.15.136:6804/2025414972 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2451364910) 0x7fe58c002c50 con 0x7fe5900739c0 2024-05-04T17:36:00.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f147d7db640 1 -- 172.21.15.110:0/2960692571 reap_dead start 2024-05-04T17:36:00.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 shutdown_connections 2024-05-04T17:36:00.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 >> 172.21.15.110:0/2960692571 conn(0x7f14700058e0 msgr2=0x7f1470036c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7f044dffb640 1 -- 172.21.15.110:0/2618623543 <== osd.5 v1:172.21.15.136:6800/2872378682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3423592458) 0x7f043c002c50 con 0x7f04400739c0 2024-05-04T17:36:00.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 shutdown_connections 2024-05-04T17:36:00.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fd8927fc640 1 -- 172.21.15.110:0/572030017 wait complete. 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 >> v1:172.21.15.136:6800/2872378682 conn(0x7f04400739c0 legacy=0x7f0440075de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 >> v1:172.21.15.110:6816/783516658 conn(0x7f044006c930 legacy=0x7f044006edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 >> v1:172.21.15.110:6789/0 conn(0x7f045c144020 legacy=0x7f045c14a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f0461b71640 1 -- 172.21.15.110:0/2618623543 reap_dead start 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 shutdown_connections 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 >> 172.21.15.110:0/2618623543 conn(0x7f045c061c70 msgr2=0x7f045c142c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 shutdown_connections 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f04635d4640 1 -- 172.21.15.110:0/2618623543 wait complete. 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 --> v1:172.21.15.136:6812/239937373 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3e40b0620 con 0x7fa3d0073bd0 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.411+0000 7fa3e0ff9640 1 -- 172.21.15.110:0/871353311 <== osd.6 v1:172.21.15.136:6812/239937373 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3371909001) 0x7fa3e40b0620 con 0x7fa3d0073bd0 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 >> v1:172.21.15.136:6812/239937373 conn(0x7fa3d0073bd0 legacy=0x7fa3d0075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 >> v1:172.21.15.110:6816/783516658 conn(0x7fa3d006cab0 legacy=0x7fa3d006ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 >> v1:172.21.15.110:6790/0 conn(0x7fa3e4071b00 legacy=0x7fa3e412aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fa3e8a50640 1 -- 172.21.15.110:0/871353311 reap_dead start 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 shutdown_connections 2024-05-04T17:36:00.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 >> 172.21.15.110:0/871353311 conn(0x7fa3e40b4210 msgr2=0x7fa3e406a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 >> v1:172.21.15.136:6804/2025414972 conn(0x7fe5900739c0 legacy=0x7fe590075de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.412+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 shutdown_connections 2024-05-04T17:36:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 shutdown_connections 2024-05-04T17:36:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fa3ea4b3640 1 -- 172.21.15.110:0/871353311 wait complete. 2024-05-04T17:36:00.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 >> v1:172.21.15.110:6816/783516658 conn(0x7fe59006c8a0 legacy=0x7fe59006ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 >> v1:172.21.15.110:6789/0 conn(0x7fe5a40cf8a0 legacy=0x7fe5a40cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7f147f23e640 1 -- 172.21.15.110:0/2960692571 wait complete. 2024-05-04T17:36:00.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b1f21640 1 -- 172.21.15.110:0/3636595386 reap_dead start 2024-05-04T17:36:00.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 shutdown_connections 2024-05-04T17:36:00.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 >> 172.21.15.110:0/3636595386 conn(0x7fe5a4005c60 msgr2=0x7fe5a40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 shutdown_connections 2024-05-04T17:36:00.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.413+0000 7fe5b2722640 1 -- 172.21.15.110:0/3636595386 wait complete. 2024-05-04T17:36:00.453 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:00.453 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-04T17:36:00.454 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2024-05-04T17:36:00.454 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-04T17:36:00.455 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2024-05-04T17:36:00.455 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-04T17:36:00.456 INFO:teuthology.orchestra.run.smithi110.stdout:60129542147 2024-05-04T17:36:00.456 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-04T17:36:00.457 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:00.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-04T17:36:00.458 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:00.458 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-04T17:36:00.459 INFO:teuthology.orchestra.run.smithi110.stdout:60129542146 2024-05-04T17:36:00.459 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-04T17:36:00.460 INFO:teuthology.orchestra.run.smithi110.stdout:60129542148 2024-05-04T17:36:00.460 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-04T17:36:00.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.657+0000 7f49de68c640 1 Processor -- start 2024-05-04T17:36:00.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.657+0000 7f49de68c640 1 -- start start 2024-05-04T17:36:00.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.657+0000 7ff6e7c45640 1 Processor -- start 2024-05-04T17:36:00.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.658+0000 7ff6e7c45640 1 -- start start 2024-05-04T17:36:00.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7ff6e61e2640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7ff6d80c8e20 0x7ff6d80d7710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46474/0 (socket says 172.21.15.110:46474) 2024-05-04T17:36:00.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7f0c2f50e640 1 Processor -- start 2024-05-04T17:36:00.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7f0c2f50e640 1 -- start start 2024-05-04T17:36:00.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7ff6e61e2640 1 -- 172.21.15.110:0/3464758812 learned_addr learned my addr 172.21.15.110:0/3464758812 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.658+0000 7f49de68c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d00ea120 con 0x7f49d0006a70 2024-05-04T17:36:00.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.658+0000 7f49de68c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d002d200 con 0x7f49d002c1a0 2024-05-04T17:36:00.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.658+0000 7f49de68c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d00e3df0 con 0x7f49d0033060 2024-05-04T17:36:00.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d802be10 con 0x7ff6d80cf870 2024-05-04T17:36:00.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7fc9e93ec640 1 Processor -- start 2024-05-04T17:36:00.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7f49dcc29640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f49d0033060 0x7f49d00f1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46460/0 (socket says 172.21.15.110:46460) 2024-05-04T17:36:00.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7f49dcc29640 1 -- 172.21.15.110:0/3181781828 learned_addr learned my addr 172.21.15.110:0/3181781828 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fb578d59640 1 Processor -- start 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fb578d59640 1 -- start start 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817227208 0 0) 0x7f49d00ea120 con 0x7f49d0006a70 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49c8003520 con 0x7f49d0006a70 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776417579 0 0) 0x7f49d00e3df0 con 0x7f49d0033060 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d00ea120 con 0x7f49d0033060 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.659+0000 7fc9e93ec640 1 -- start start 2024-05-04T17:36:00.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fc9e93ec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40afeb0 con 0x7fc9e4141a30 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fc9e93ec640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e4070340 con 0x7fc9e413bb40 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fc9e93ec640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e4070620 con 0x7fc9e406f800 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f0c2f50e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c20036430 con 0x7f0c200389e0 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f0c2f50e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c20039840 con 0x7f0c20030660 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f0c2f50e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c20035690 con 0x7f0c20038dc0 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f0c2ed0d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0c20030660 0x7f0c200d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37154/0 (socket says 172.21.15.110:37154) 2024-05-04T17:36:00.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f0c2ed0d640 1 -- 172.21.15.110:0/3696124193 learned_addr learned my addr 172.21.15.110:0/3696124193 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f3e59c2a640 1 Processor -- start 2024-05-04T17:36:00.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7f3e59c2a640 1 -- start start 2024-05-04T17:36:00.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fc9e8beb640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fc9e406f800 0x7fc9e4149b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46496/0 (socket says 172.21.15.110:46496) 2024-05-04T17:36:00.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.660+0000 7fc9e8beb640 1 -- 172.21.15.110:0/3329717164 learned_addr learned my addr 172.21.15.110:0/3329717164 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f0c5b59e640 1 Processor -- start 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d802c0f0 con 0x7ff6d8006da0 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2151954014 0 0) 0x7f49d002d200 con 0x7f49d002c1a0 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d00e3df0 con 0x7f49d002c1a0 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3150845007 0 0) 0x7f49c8003520 con 0x7f49d0006a70 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 >> v1:172.21.15.110:6790/0 conn(0x7f49d0033060 legacy=0x7f49d00f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 >> v1:172.21.15.136:6789/0 conn(0x7f49d002c1a0 legacy=0x7f49d00e36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d00e3fc0 con 0x7f49d0006a70 2024-05-04T17:36:00.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f49cc001590 con 0x7f49d0006a70 2024-05-04T17:36:00.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f0c5b59e640 1 -- start start 2024-05-04T17:36:00.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 408591402 0 0) 0x7fc9e4070620 con 0x7fc9e406f800 2024-05-04T17:36:00.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9d0003520 con 0x7fc9e406f800 2024-05-04T17:36:00.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2445775438 0 0) 0x7fc9e4070340 con 0x7fc9e413bb40 2024-05-04T17:36:00.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e4070620 con 0x7fc9e413bb40 2024-05-04T17:36:00.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2441868031 0 0) 0x7f0c18001110 con 0x7f0c200389e0 2024-05-04T17:36:00.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c14003520 con 0x7f0c200389e0 2024-05-04T17:36:00.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791383945 0 0) 0x7f0c20039840 con 0x7f0c20030660 2024-05-04T17:36:00.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c18001110 con 0x7f0c20030660 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49cc0017f0 con 0x7f49d0006a70 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f49d6ffd640 1 -- 172.21.15.110:0/3181781828 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f49cc001bc0 con 0x7f49d0006a70 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2195338514 0 0) 0x7fc9e40afeb0 con 0x7fc9e4141a30 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e4070340 con 0x7fc9e4141a30 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 359293727 0 0) 0x7fc9e4070620 con 0x7fc9e413bb40 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 >> v1:172.21.15.110:6790/0 conn(0x7fc9e406f800 legacy=0x7fc9e4149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 >> v1:172.21.15.110:6789/0 conn(0x7fc9e4141a30 legacy=0x7fc9e413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e4070900 con 0x7fc9e413bb40 2024-05-04T17:36:00.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc9dc001570 con 0x7fc9e413bb40 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9dc0017d0 con 0x7fc9e413bb40 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fc9e2ffd640 1 -- 172.21.15.110:0/3329717164 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc9dc001c20 con 0x7fc9e413bb40 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.661+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d802c3d0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066433615 0 0) 0x7ff6d802be10 con 0x7ff6d80cf870 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6cc003520 con 0x7ff6d80cf870 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773302618 0 0) 0x7ff6d802c0f0 con 0x7ff6d8006da0 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d802be10 con 0x7ff6d8006da0 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717210832 0 0) 0x7ff6d802c3d0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d802c0f0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1651889484 0 0) 0x7ff6cc003520 con 0x7ff6d80cf870 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 >> v1:172.21.15.110:6790/0 conn(0x7ff6d80c8e20 legacy=0x7ff6d80d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 >> v1:172.21.15.136:6789/0 conn(0x7ff6d8006da0 legacy=0x7ff6d80cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d802b1c0 con 0x7ff6d80cf870 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff6d0001590 con 0x7ff6d80cf870 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff6d00017f0 con 0x7ff6d80cf870 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7ff6e49df640 1 -- 172.21.15.110:0/3464758812 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff6d0001b40 con 0x7ff6d80cf870 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fb571d9b640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb57406ff00 0x7fb5741424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46502/0 (socket says 172.21.15.110:46502) 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fb571d9b640 1 -- 172.21.15.110:0/3523953718 learned_addr learned my addr 172.21.15.110:0/3523953718 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb574072050 con 0x7fb57406fb20 2024-05-04T17:36:00.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb574072c20 con 0x7fb574070c70 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fc909e74640 1 Processor -- start 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f0c5b59e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c071bf0 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f0c53fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0c5c070240 0x7f0c5c13e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46510/0 (socket says 172.21.15.110:46510) 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f0c53fff640 1 -- 172.21.15.110:0/3505804279 learned_addr learned my addr 172.21.15.110:0/3505804279 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c06e820 con 0x7f0c5c06f680 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c5ad9d640 1 -- 172.21.15.110:0/3505804279 >> v1:172.21.15.136:6789/0 conn(0x7f0c5c06f680 legacy=0x7f0c5c149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-04T17:36:00.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c5ad9d640 1 -- 172.21.15.110:0/3505804279 >> v1:172.21.15.136:6789/0 conn(0x7f0c5c06f680 legacy=0x7f0c5c149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c5ad9d640 1 --1- 172.21.15.110:0/3505804279 >> v1:172.21.15.136:6789/0 conn(0x7f0c5c06f680 0x7f0c5c149e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f3e59c2a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4c02d220 con 0x7f3e4c0074f0 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f3e59429640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f3e4c0c4c40 0x7f3e4c0cfe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46508/0 (socket says 172.21.15.110:46508) 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f3e59429640 1 -- 172.21.15.110:0/4236628611 learned_addr learned my addr 172.21.15.110:0/4236628611 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4c02d500 con 0x7f3e4c0078d0 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4c0c2070 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c13c080 con 0x7f0c5c070240 2024-05-04T17:36:00.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f49de68c640 1 -- 172.21.15.110:0/3181781828 >> v1:172.21.15.110:6789/0 conn(0x7f49d0006a70 legacy=0x7f49d00e9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4070355806 0 0) 0x7f0c14003520 con 0x7f0c200389e0 2024-05-04T17:36:00.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 >> v1:172.21.15.110:6790/0 conn(0x7f0c20038dc0 legacy=0x7f0c200cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 >> v1:172.21.15.136:6789/0 conn(0x7f0c20030660 legacy=0x7f0c200d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c20035f10 con 0x7f0c200389e0 2024-05-04T17:36:00.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c180017e0 con 0x7f0c200389e0 2024-05-04T17:36:00.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.662+0000 7fc909e74640 1 -- start start 2024-05-04T17:36:00.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076746962 0 0) 0x7f3e4c0c2070 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e44003520 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f49de68c640 1 -- 172.21.15.110:0/3181781828 shutdown_connections 2024-05-04T17:36:00.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f49de68c640 1 -- 172.21.15.110:0/3181781828 >> 172.21.15.110:0/3181781828 conn(0x7f49d00058f0 msgr2=0x7f49d0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5740af190 con 0x7fb57406ff00 2024-05-04T17:36:00.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f49de68c640 1 -- 172.21.15.110:0/3181781828 shutdown_connections 2024-05-04T17:36:00.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc9e93ec640 1 -- 172.21.15.110:0/3329717164 >> v1:172.21.15.136:6789/0 conn(0x7fc9e413bb40 legacy=0x7fc9e406f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410047687 0 0) 0x7f3e4c02d500 con 0x7f3e4c0078d0 2024-05-04T17:36:00.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e4c0c2070 con 0x7f3e4c0078d0 2024-05-04T17:36:00.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc909e74640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90413e7f0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc909e74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc904071ae0 con 0x7fc9040aec70 2024-05-04T17:36:00.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc909e74640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90406fb20 con 0x7fc90406f340 2024-05-04T17:36:00.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc9e93ec640 1 -- 172.21.15.110:0/3329717164 shutdown_connections 2024-05-04T17:36:00.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc9e93ec640 1 -- 172.21.15.110:0/3329717164 >> 172.21.15.110:0/3329717164 conn(0x7fc9e40b3a70 msgr2=0x7fc9e40b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc9037fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fc90406f340 0x7fc90406fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46526/0 (socket says 172.21.15.110:46526) 2024-05-04T17:36:00.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc903fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc9040aec70 0x7fc904072370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37202/0 (socket says 172.21.15.110:37202) 2024-05-04T17:36:00.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc903fff640 1 -- 172.21.15.110:0/384041789 learned_addr learned my addr 172.21.15.110:0/384041789 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 >> v1:172.21.15.110:6789/0 conn(0x7ff6d80cf870 legacy=0x7ff6d80c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1425678213 0 0) 0x7f3e4c0c2070 con 0x7f3e4c0078d0 2024-05-04T17:36:00.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc9e93ec640 1 -- 172.21.15.110:0/3329717164 shutdown_connections 2024-05-04T17:36:00.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 >> v1:172.21.15.110:6790/0 conn(0x7f3e4c0c4c40 legacy=0x7f3e4c0cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 >> v1:172.21.15.110:6789/0 conn(0x7f3e4c0074f0 legacy=0x7f3e4c0be2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:36:00.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c02b650 con 0x7f3e4c0078d0 2024-05-04T17:36:00.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3e40001590 con 0x7f3e4c0078d0 2024-05-04T17:36:00.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 shutdown_connections 2024-05-04T17:36:00.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 >> 172.21.15.110:0/3464758812 conn(0x7ff6d8005c20 msgr2=0x7ff6d8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103123222 0 0) 0x7fb574072c20 con 0x7fb574070c70 2024-05-04T17:36:00.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb55c003520 con 0x7fb574070c70 2024-05-04T17:36:00.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc9e93ec640 1 -- 172.21.15.110:0/3329717164 wait complete. 2024-05-04T17:36:00.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e400017f0 con 0x7f3e4c0078d0 2024-05-04T17:36:00.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16273838 0 0) 0x7f0c5c13c080 con 0x7f0c5c070240 2024-05-04T17:36:00.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c48003440 con 0x7f0c5c070240 2024-05-04T17:36:00.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f3e537fe640 1 -- 172.21.15.110:0/4236628611 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3e40001c40 con 0x7f3e4c0078d0 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1846531940 0 0) 0x7fb55c003520 con 0x7fb574070c70 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 >> v1:172.21.15.110:6790/0 conn(0x7fb57406ff00 legacy=0x7fb5741424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 >> v1:172.21.15.110:6789/0 conn(0x7fb57406fb20 legacy=0x7fb5740aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb574073140 con 0x7fb574070c70 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb56c001990 con 0x7fb574070c70 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4226160824 0 0) 0x7fc904071ae0 con 0x7fc9040aec70 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8ec003520 con 0x7fc9040aec70 2024-05-04T17:36:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7fc902ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc9040ae7f0 0x7fc90413bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36240/0 (socket says 172.21.15.110:36240) 2024-05-04T17:36:00.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790650633 0 0) 0x7f0c5c071bf0 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c5c13c080 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fc9e93ec640 1 Processor -- start 2024-05-04T17:36:00.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 shutdown_connections 2024-05-04T17:36:00.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7ff6e7c45640 1 -- 172.21.15.110:0/3464758812 wait complete. 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb56c001bf0 con 0x7fb574070c70 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fb57159a640 1 -- 172.21.15.110:0/3523953718 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb56c002000 con 0x7fb574070c70 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 669495519 0 0) 0x7fc8ec003520 con 0x7fc9040aec70 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 >> v1:172.21.15.110:6790/0 conn(0x7fc90406f340 legacy=0x7fc90406fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 >> v1:172.21.15.110:6789/0 conn(0x7fc9040ae7f0 legacy=0x7fc90413bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc904072a80 con 0x7fc9040aec70 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc8fc001970 con 0x7fc9040aec70 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8fc001ca0 con 0x7fc9040aec70 2024-05-04T17:36:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7fc9027fc640 1 -- 172.21.15.110:0/384041789 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc8fc0017d0 con 0x7fc9040aec70 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2410919149 0 0) 0x7f0c5c13c080 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 >> v1:172.21.15.110:6790/0 conn(0x7f0c5c070240 legacy=0x7f0c5c13e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 >> v1:172.21.15.136:6789/0 conn(0x7f0c5c06f680 legacy=0x7f0c5c149e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c13f010 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c4c0015f0 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 >> v1:172.21.15.136:6789/0 conn(0x7f3e4c0078d0 legacy=0x7f3e4c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.663+0000 7f49de68c640 1 -- 172.21.15.110:0/3181781828 wait complete. 2024-05-04T17:36:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f49de68c640 1 Processor -- start 2024-05-04T17:36:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f49de68c640 1 -- start start 2024-05-04T17:36:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 >> v1:172.21.15.136:6789/0 conn(0x7fb574070c70 legacy=0x7fb57414a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.664+0000 7f0c2d50a640 1 -- 172.21.15.110:0/3696124193 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c18001a40 con 0x7f0c200389e0 2024-05-04T17:36:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3696124193 >> v1:172.21.15.110:6789/0 conn(0x7f0c200389e0 legacy=0x7f0c20033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3696124193 shutdown_connections 2024-05-04T17:36:00.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3696124193 >> 172.21.15.110:0/3696124193 conn(0x7f0c20005c00 msgr2=0x7f0c20006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3696124193 shutdown_connections 2024-05-04T17:36:00.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c4c001910 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 shutdown_connections 2024-05-04T17:36:00.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c59d9b640 1 -- 172.21.15.110:0/3505804279 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c4c001450 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 >> 172.21.15.110:0/4236628611 conn(0x7f3e4c006530 msgr2=0x7f3e4c006940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7ff6e7c45640 1 Processor -- start 2024-05-04T17:36:00.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 shutdown_connections 2024-05-04T17:36:00.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 >> 172.21.15.110:0/3523953718 conn(0x7fb5740b4af0 msgr2=0x7fb5740b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fc909e74640 1 -- 172.21.15.110:0/384041789 >> v1:172.21.15.136:6789/0 conn(0x7fc9040aec70 legacy=0x7fc904072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7ff6e7c45640 1 -- start start 2024-05-04T17:36:00.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3696124193 wait complete. 2024-05-04T17:36:00.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 shutdown_connections 2024-05-04T17:36:00.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fc909e74640 1 -- 172.21.15.110:0/384041789 shutdown_connections 2024-05-04T17:36:00.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fc909e74640 1 -- 172.21.15.110:0/384041789 >> 172.21.15.110:0/384041789 conn(0x7fc9040b4210 msgr2=0x7fc9040b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 shutdown_connections 2024-05-04T17:36:00.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fc909e74640 1 -- 172.21.15.110:0/384041789 shutdown_connections 2024-05-04T17:36:00.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7f3e59c2a640 1 -- 172.21.15.110:0/4236628611 wait complete. 2024-05-04T17:36:00.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 >> v1:172.21.15.110:6789/0 conn(0x7f0c5c06f2a0 legacy=0x7f0c5c13b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7fc9e93ec640 1 -- start start 2024-05-04T17:36:00.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7f49d77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f49d00e4290 0x7f49d017e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36242/0 (socket says 172.21.15.110:36242) 2024-05-04T17:36:00.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7f49d7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f49d0006a70 0x7f49d00e91c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37208/0 (socket says 172.21.15.110:37208) 2024-05-04T17:36:00.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7f49d77fe640 1 -- 172.21.15.110:0/1646282154 learned_addr learned my addr 172.21.15.110:0/1646282154 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d017dde0 con 0x7f49d00e4290 2024-05-04T17:36:00.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7ff6e51e0640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7ff6d80c8e20 0x7ff6d80cd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46534/0 (socket says 172.21.15.110:46534) 2024-05-04T17:36:00.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7ff6e59e1640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff6d8006da0 0x7ff6d8164780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37222/0 (socket says 172.21.15.110:37222) 2024-05-04T17:36:00.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7ff6e51e0640 1 -- 172.21.15.110:0/290470202 learned_addr learned my addr 172.21.15.110:0/290470202 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d802f930 con 0x7ff6d80cf870 2024-05-04T17:36:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d80c3650 con 0x7ff6d8006da0 2024-05-04T17:36:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.665+0000 7fb578d59640 1 -- 172.21.15.110:0/3523953718 wait complete. 2024-05-04T17:36:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7fc9e93ec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e41e4210 con 0x7fc9e4141a30 2024-05-04T17:36:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7fc9e93ec640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40ac120 con 0x7fc9e413bb40 2024-05-04T17:36:00.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7fc9e93ec640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40ab360 con 0x7fc9e406f800 2024-05-04T17:36:00.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 shutdown_connections 2024-05-04T17:36:00.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 >> 172.21.15.110:0/3505804279 conn(0x7f0c5c0b4120 msgr2=0x7f0c5c0b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:00.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d80cdff0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.666+0000 7fc909e74640 1 -- 172.21.15.110:0/384041789 wait complete. 2024-05-04T17:36:00.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fc909e74640 1 Processor -- start 2024-05-04T17:36:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fb578d59640 1 Processor -- start 2024-05-04T17:36:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 shutdown_connections 2024-05-04T17:36:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fc9e3fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fc9e406f800 0x7fc9e41e3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46560/0 (socket says 172.21.15.110:46560) 2024-05-04T17:36:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fc9e3fff640 1 -- 172.21.15.110:0/212154507 learned_addr learned my addr 172.21.15.110:0/212154507 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636059176 0 0) 0x7ff6d80cdff0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fb578d59640 1 -- start start 2024-05-04T17:36:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f3e59c2a640 1 Processor -- start 2024-05-04T17:36:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f0c5b59e640 1 -- 172.21.15.110:0/3505804279 wait complete. 2024-05-04T17:36:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c4003520 con 0x7ff6d80c8e20 2024-05-04T17:36:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f0c2f50e640 1 Processor -- start 2024-05-04T17:36:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7f0c2f50e640 1 -- start start 2024-05-04T17:36:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3601903420 0 0) 0x7ff6d80c3650 con 0x7ff6d8006da0 2024-05-04T17:36:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d80cdff0 con 0x7ff6d8006da0 2024-05-04T17:36:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2000777354 0 0) 0x7fc9e40ac120 con 0x7fc9e413bb40 2024-05-04T17:36:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7f3e59c2a640 1 -- start start 2024-05-04T17:36:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c4003520 con 0x7fc9e413bb40 2024-05-04T17:36:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7f0c2f50e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c200bfe60 con 0x7f0c2000d7e0 2024-05-04T17:36:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7f0c2f50e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c20169740 con 0x7f0c20030660 2024-05-04T17:36:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7f0c2f50e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2000cf90 con 0x7f0c200389e0 2024-05-04T17:36:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2906681944 0 0) 0x7ff6c4003520 con 0x7ff6d80c8e20 2024-05-04T17:36:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d80c3650 con 0x7ff6d80c8e20 2024-05-04T17:36:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff6d00035f0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fb578d59640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5740adf50 con 0x7fb574070c70 2024-05-04T17:36:00.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fb578d59640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57413f0f0 con 0x7fb57406ff00 2024-05-04T17:36:00.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fb578d59640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57413f2c0 con 0x7fb57406fb20 2024-05-04T17:36:00.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1614080569 0 0) 0x7fc9e40ab360 con 0x7fc9e406f800 2024-05-04T17:36:00.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e40ac120 con 0x7fc9e406f800 2024-05-04T17:36:00.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1630750868 0 0) 0x7fc9c4003520 con 0x7fc9e413bb40 2024-05-04T17:36:00.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e40ab360 con 0x7fc9e413bb40 2024-05-04T17:36:00.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc9dc001a80 con 0x7fc9e413bb40 2024-05-04T17:36:00.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1019656939 0 0) 0x7ff6d80cdff0 con 0x7ff6d8006da0 2024-05-04T17:36:00.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6c4003520 con 0x7ff6d8006da0 2024-05-04T17:36:00.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff6dc001780 con 0x7ff6d8006da0 2024-05-04T17:36:00.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.667+0000 7fc909e74640 1 -- start start 2024-05-04T17:36:00.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9037fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc90406f340 0x7fc9041e4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36270/0 (socket says 172.21.15.110:36270) 2024-05-04T17:36:00.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fb57259c640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb57406fb20 0x7fb5740ad6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46566/0 (socket says 172.21.15.110:46566) 2024-05-04T17:36:00.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fb57259c640 1 -- 172.21.15.110:0/4248935177 learned_addr learned my addr 172.21.15.110:0/4248935177 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 410339712 0 0) 0x7fc9e41e4210 con 0x7fc9e4141a30 2024-05-04T17:36:00.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c4003520 con 0x7fc9e4141a30 2024-05-04T17:36:00.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc909e74640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9041e44c0 con 0x7fc90406f340 2024-05-04T17:36:00.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc909e74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90414f6e0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7fc9037fe640 1 -- 172.21.15.110:0/434846807 learned_addr learned my addr 172.21.15.110:0/434846807 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079250502 0 0) 0x7fb57413f0f0 con 0x7fb57406ff00 2024-05-04T17:36:00.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb554003520 con 0x7fb57406ff00 2024-05-04T17:36:00.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9040b2ca0 con 0x7fc9040aec70 2024-05-04T17:36:00.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 940568357 0 0) 0x7ff6d80c3650 con 0x7ff6d80c8e20 2024-05-04T17:36:00.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d00e5710 con 0x7f49d0006a70 2024-05-04T17:36:00.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d00e98d0 con 0x7f49d017e9d0 2024-05-04T17:36:00.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 >> v1:172.21.15.136:6789/0 conn(0x7ff6d8006da0 legacy=0x7ff6d8164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836211228 0 0) 0x7f49d00e5710 con 0x7f49d0006a70 2024-05-04T17:36:00.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49bc003520 con 0x7f49d0006a70 2024-05-04T17:36:00.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006531059 0 0) 0x7f49d017dde0 con 0x7f49d00e4290 2024-05-04T17:36:00.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d00e5710 con 0x7f49d00e4290 2024-05-04T17:36:00.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 384148588 0 0) 0x7fc9c4003520 con 0x7fc9e4141a30 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e41e4210 con 0x7fc9e4141a30 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 136449633 0 0) 0x7fc9e40ab360 con 0x7fc9e413bb40 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 >> v1:172.21.15.110:6790/0 conn(0x7fc9e406f800 legacy=0x7fc9e41e3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 >> v1:172.21.15.110:6789/0 conn(0x7fc9e4141a30 legacy=0x7fc9e41ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e40aa3a0 con 0x7fc9e413bb40 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e93ec640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e40aa690 con 0x7fc9e413bb40 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157403711 0 0) 0x7fc90414f6e0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8f0003520 con 0x7fc9040ae7f0 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414180619 0 0) 0x7fc9041e44c0 con 0x7fc90406f340 2024-05-04T17:36:00.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc90414f6e0 con 0x7fc90406f340 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f3e53fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3e4c0c4c40 0x7f3e4c0b3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37250/0 (socket says 172.21.15.110:37250) 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f3e59429640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3e4c0b3b60 0x7f3e4c164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36272/0 (socket says 172.21.15.110:36272) 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f3e59429640 1 -- 172.21.15.110:0/791423158 learned_addr learned my addr 172.21.15.110:0/791423158 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4c0b6120 con 0x7f3e4c0b3b60 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102039917 0 0) 0x7fb57413f2c0 con 0x7fb57406fb20 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb57413f0f0 con 0x7fb57406fb20 2024-05-04T17:36:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3485816992 0 0) 0x7fb554003520 con 0x7fb57406ff00 2024-05-04T17:36:00.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb57413f2c0 con 0x7fb57406ff00 2024-05-04T17:36:00.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566903520 0 0) 0x7fb5740adf50 con 0x7fb574070c70 2024-05-04T17:36:00.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4c0b54f0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc9e93ec640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9e40ad150 con 0x7fc9e413bb40 2024-05-04T17:36:00.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb554003520 con 0x7fb574070c70 2024-05-04T17:36:00.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb5680017a0 con 0x7fb57406ff00 2024-05-04T17:36:00.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4c0b6640 con 0x7f3e4c0074f0 2024-05-04T17:36:00.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f0c5b59e640 1 Processor -- start 2024-05-04T17:36:00.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f0c5b59e640 1 -- start start 2024-05-04T17:36:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371261920 0 0) 0x7f49d00e98d0 con 0x7f49d017e9d0 2024-05-04T17:36:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d017dde0 con 0x7f49d017e9d0 2024-05-04T17:36:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2467309891 0 0) 0x7f49bc003520 con 0x7f49d0006a70 2024-05-04T17:36:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49d00e98d0 con 0x7f49d0006a70 2024-05-04T17:36:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f49cc001b40 con 0x7f49d0006a70 2024-05-04T17:36:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3198605698 0 0) 0x7f49d00e5710 con 0x7f49d00e4290 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49bc003520 con 0x7f49d00e4290 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f49c40017e0 con 0x7f49d00e4290 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364451701 0 0) 0x7fc9040b2ca0 con 0x7fc9040aec70 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9041e44c0 con 0x7fc9040aec70 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 575509748 0 0) 0x7fc8f0003520 con 0x7fc9040ae7f0 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9040b2ca0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc8f80014d0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1245782480 0 0) 0x7f49d00e98d0 con 0x7f49d0006a70 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 >> v1:172.21.15.110:6790/0 conn(0x7f49d017e9d0 legacy=0x7f49d00dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 >> v1:172.21.15.110:6789/0 conn(0x7f49d00e4290 legacy=0x7f49d017e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d0033db0 con 0x7f49d0006a70 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49d00db4f0 con 0x7f49d0006a70 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f49d01870f0 con 0x7f49d0006a70 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9dc001790 con 0x7fc9e413bb40 2024-05-04T17:36:00.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc9dc003ae0 con 0x7fc9e413bb40 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 >> v1:172.21.15.110:6789/0 conn(0x7ff6d80cf870 legacy=0x7ff6d80c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d80c31b0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d8165170 con 0x7ff6d80c8e20 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6d802e4c0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff6d0003b00 con 0x7ff6d80c8e20 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff6d0003ed0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.668+0000 7f0c2ed0d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0c2000d7e0 0x7f0c20174c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36260/0 (socket says 172.21.15.110:36260) 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.669+0000 7f0c2ed0d640 1 -- 172.21.15.110:0/3034316308 learned_addr learned my addr 172.21.15.110:0/3034316308 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2111824667 0 0) 0x7f0c200bfe60 con 0x7f0c2000d7e0 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c0c003520 con 0x7f0c2000d7e0 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4098802159 0 0) 0x7f0c20169740 con 0x7f0c20030660 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c200bfe60 con 0x7f0c20030660 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1685727389 0 0) 0x7fc9040b2ca0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 >> v1:172.21.15.110:6790/0 conn(0x7fc9040aec70 legacy=0x7fc9041e3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 >> v1:172.21.15.110:6789/0 conn(0x7fc90406f340 legacy=0x7fc9041e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9040b2e70 con 0x7fc9040ae7f0 2024-05-04T17:36:00.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc904141510 con 0x7fc9040ae7f0 2024-05-04T17:36:00.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9041e5120 con 0x7fc9040ae7f0 2024-05-04T17:36:00.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c5b59e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c134ac0 con 0x7f0c5c06f680 2024-05-04T17:36:00.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c5b59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c0b3ae0 con 0x7f0c5c1d3c50 2024-05-04T17:36:00.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f0c5b59e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c139400 con 0x7f0c5c06f2a0 2024-05-04T17:36:00.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310350010 0 0) 0x7f3e4c0b6120 con 0x7f3e4c0b3b60 2024-05-04T17:36:00.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.670+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e38003520 con 0x7f3e4c0b3b60 2024-05-04T17:36:00.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394470509 0 0) 0x7f3e4c0b54f0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e4c0b6120 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d0004cd0 con 0x7f49d0006a70 2024-05-04T17:36:00.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49cc0017f0 con 0x7f49d0006a70 2024-05-04T17:36:00.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f49cc003a40 con 0x7f49d0006a70 2024-05-04T17:36:00.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d80c3440 con 0x7ff6d80c8e20 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1321210399 0 0) 0x7f0c200bfe60 con 0x7f0c20030660 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c20169740 con 0x7f0c20030660 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c180016e0 con 0x7f0c20030660 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3145876942 0 0) 0x7f0c20169740 con 0x7f0c20030660 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 >> v1:172.21.15.110:6790/0 conn(0x7f0c200389e0 legacy=0x7f0c20166590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 >> v1:172.21.15.110:6789/0 conn(0x7f0c2000d7e0 legacy=0x7f0c20174c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c201693f0 con 0x7f0c20030660 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2314708069 0 0) 0x7fb57413f2c0 con 0x7fb57406ff00 2024-05-04T17:36:00.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 >> v1:172.21.15.110:6790/0 conn(0x7fb57406fb20 legacy=0x7fb5740ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 >> v1:172.21.15.110:6789/0 conn(0x7fb574070c70 legacy=0x7fb5741eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5740aade0 con 0x7fb57406ff00 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5740ab0d0 con 0x7fb57406ff00 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5740abcd0 con 0x7fb57406ff00 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb568001cd0 con 0x7fb57406ff00 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb568004980 con 0x7fb57406ff00 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8f8001af0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc8f8001330 con 0x7fc9040ae7f0 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f0c53fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0c5c06f680 0x7f0c5c133ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36286/0 (socket says 172.21.15.110:36286) 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f0c53fff640 1 -- 172.21.15.110:0/1592742643 learned_addr learned my addr 172.21.15.110:0/1592742643 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 11772945 0 0) 0x7f3e4c0b6120 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e4c0b54f0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.671+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3e40001b40 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766399543 0 0) 0x7f3e4c0b6640 con 0x7f3e4c0074f0 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e4c0b6120 con 0x7f3e4c0074f0 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4039576942 0 0) 0x7f3e38003520 con 0x7f3e4c0b3b60 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e4c0b6640 con 0x7f3e4c0b3b60 2024-05-04T17:36:00.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3e5406a050 con 0x7f3e4c0b3b60 2024-05-04T17:36:00.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c20166ca0 con 0x7f0c20030660 2024-05-04T17:36:00.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c200be720 con 0x7f0c20030660 2024-05-04T17:36:00.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.672+0000 7fc9e93ec640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c8005080 con 0x7fc9e413bb40 2024-05-04T17:36:00.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fc9dc019c50 con 0x7fc9e413bb40 2024-05-04T17:36:00.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c18001a40 con 0x7f0c20030660 2024-05-04T17:36:00.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c18001540 con 0x7f0c20030660 2024-05-04T17:36:00.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb558005080 con 0x7fb57406ff00 2024-05-04T17:36:00.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1726387238 0 0) 0x7f0c5c134ac0 con 0x7f0c5c06f680 2024-05-04T17:36:00.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c3c003520 con 0x7f0c5c06f680 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007988379 0 0) 0x7f0c5c0b3ae0 con 0x7f0c5c1d3c50 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c5c134ac0 con 0x7f0c5c1d3c50 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1289374860 0 0) 0x7f3e4c0b54f0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 >> v1:172.21.15.110:6790/0 conn(0x7f3e4c0074f0 legacy=0x7f3e4c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 >> v1:172.21.15.110:6789/0 conn(0x7f3e4c0b3b60 legacy=0x7f3e4c164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c0b6930 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fc9dc084cd0 con 0x7fc9e413bb40 2024-05-04T17:36:00.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fc8f801ac10 con 0x7fc9040ae7f0 2024-05-04T17:36:00.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90404fe40 con 0x7fc9040ae7f0 2024-05-04T17:36:00.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c0b2f30 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.673+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e4c0b5d40 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f49cc003c60 con 0x7f49d0006a70 2024-05-04T17:36:00.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fc8f8085d20 con 0x7fc9040ae7f0 2024-05-04T17:36:00.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2219510236 0 0) 0x7f0c3c003520 con 0x7f0c5c06f680 2024-05-04T17:36:00.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c5c0b3ae0 con 0x7f0c5c06f680 2024-05-04T17:36:00.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c440017e0 con 0x7f0c5c06f680 2024-05-04T17:36:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c10005080 con 0x7f0c20030660 2024-05-04T17:36:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4c0b81c0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3177998105 0 0) 0x7f0c5c0b3ae0 con 0x7f0c5c06f680 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 >> v1:172.21.15.110:6790/0 conn(0x7f0c5c06f2a0 legacy=0x7f0c5c1d3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 >> v1:172.21.15.136:6789/0 conn(0x7f0c5c1d3c50 legacy=0x7f0c5c1343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c141de0 con 0x7f0c5c06f680 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.674+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c13b370 con 0x7f0c5c06f680 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.675+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c440047e0 con 0x7f0c5c06f680 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.675+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c5c13e010 con 0x7f0c5c06f680 2024-05-04T17:36:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.675+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0c44004b30 con 0x7f0c5c06f680 2024-05-04T17:36:00.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.675+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f49cc086040 con 0x7f49d0006a70 2024-05-04T17:36:00.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.676+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5c04fb10 con 0x7f0c5c06f680 2024-05-04T17:36:00.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.676+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7ff6d0004070 con 0x7ff6d80c8e20 2024-05-04T17:36:00.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.676+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fb568004980 con 0x7fb57406ff00 2024-05-04T17:36:00.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.676+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fb568085f10 con 0x7fb57406ff00 2024-05-04T17:36:00.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.677+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7ff6d0086210 con 0x7ff6d80c8e20 2024-05-04T17:36:00.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.677+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f0c44004d50 con 0x7f0c5c06f680 2024-05-04T17:36:00.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.677+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f0c440861e0 con 0x7f0c5c06f680 2024-05-04T17:36:00.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.678+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e400017f0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.678+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3e40003ac0 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.678+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fc9dc053b60 con 0x7fc9e413bb40 2024-05-04T17:36:00.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.680+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7ff6d00550a0 con 0x7ff6d80c8e20 2024-05-04T17:36:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.681+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f0c440550d0 con 0x7f0c5c06f680 2024-05-04T17:36:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.682+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f49cc054ed0 con 0x7f49d0006a70 2024-05-04T17:36:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.684+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f0c18003c10 con 0x7f0c20030660 2024-05-04T17:36:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.688+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fb568054da0 con 0x7fb57406ff00 2024-05-04T17:36:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.691+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fc8f8054cf0 con 0x7fc9040ae7f0 2024-05-04T17:36:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.692+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f0c180513e0 con 0x7f0c20030660 2024-05-04T17:36:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.693+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f3e40003c60 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.693+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f3e40086060 con 0x7f3e4c0c4c40 2024-05-04T17:36:00.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.696+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f0c20036430 con 0x7f0c20030660 2024-05-04T17:36:00.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:00.700+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f3e40054ef0 con 0x7f3e4c0c4c40 2024-05-04T17:36:01.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.084+0000 7fc9e93ec640 1 -- 172.21.15.110:0/212154507 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc9c8005370 con 0x7fc9e413bb40 2024-05-04T17:36:01.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.084+0000 7fc9e0ff9640 1 -- 172.21.15.110:0/212154507 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 2439918515) 0x7fc9dc053460 con 0x7fc9e413bb40 2024-05-04T17:36:01.086 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:01.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.086+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff6d8004cd0 con 0x7ff6d80c8e20 2024-05-04T17:36:01.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.087+0000 7ff6d67fc640 1 -- 172.21.15.110:0/290470202 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 2439918515) 0x7ff6d00549a0 con 0x7ff6d80c8e20 2024-05-04T17:36:01.089 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:01.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.088+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f0c5c06fe10 con 0x7f0c5c06f680 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.088+0000 7fc9da7fc640 1 -- 172.21.15.110:0/212154507 >> v1:172.21.15.110:6816/783516658 conn(0x7fc9c406c780 legacy=0x7fc9c406ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.088+0000 7fc9da7fc640 1 -- 172.21.15.110:0/212154507 >> v1:172.21.15.136:6789/0 conn(0x7fc9e413bb40 legacy=0x7fc9e40aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.088+0000 7fc9da7fc640 1 -- 172.21.15.110:0/212154507 shutdown_connections 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.088+0000 7fc9da7fc640 1 -- 172.21.15.110:0/212154507 >> 172.21.15.110:0/212154507 conn(0x7fc9e40b3a70 msgr2=0x7fc9e406a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.089+0000 7fc9da7fc640 1 -- 172.21.15.110:0/212154507 shutdown_connections 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.089+0000 7fc9da7fc640 1 -- 172.21.15.110:0/212154507 wait complete. 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.088+0000 7f0c52ffd640 1 -- 172.21.15.110:0/1592742643 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (unknown 1482059429 0 663305998) 0x7f0c440549d0 con 0x7f0c5c06f680 2024-05-04T17:36:01.090 INFO:teuthology.orchestra.run.smithi110.stdout:60129542145 2024-05-04T17:36:01.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.091+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 >> v1:172.21.15.110:6816/783516658 conn(0x7ff6c406c7b0 legacy=0x7ff6c406ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.091+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 >> v1:172.21.15.110:6790/0 conn(0x7ff6d80c8e20 legacy=0x7ff6d80cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.091+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 shutdown_connections 2024-05-04T17:36:01.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.091+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 >> 172.21.15.110:0/290470202 conn(0x7ff6d8005c20 msgr2=0x7ff6d80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.092+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 shutdown_connections 2024-05-04T17:36:01.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.092+0000 7ff6e7c45640 1 -- 172.21.15.110:0/290470202 wait complete. 2024-05-04T17:36:01.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.093+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 >> v1:172.21.15.110:6816/783516658 conn(0x7f0c3c06c920 legacy=0x7f0c3c06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.094+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 >> v1:172.21.15.110:6789/0 conn(0x7f0c5c06f680 legacy=0x7f0c5c133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.094+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 shutdown_connections 2024-05-04T17:36:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.094+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 >> 172.21.15.110:0/1592742643 conn(0x7f0c5c0b4120 msgr2=0x7f0c5c0af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.094+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 shutdown_connections 2024-05-04T17:36:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.094+0000 7f0c5b59e640 1 -- 172.21.15.110:0/1592742643 wait complete. 2024-05-04T17:36:01.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.098+0000 7f49de68c640 1 -- 172.21.15.110:0/1646282154 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f49d00e4670 con 0x7f49d0006a70 2024-05-04T17:36:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.099+0000 7f49d4ff9640 1 -- 172.21.15.110:0/1646282154 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 3130653186) 0x7f49cc0547d0 con 0x7f49d0006a70 2024-05-04T17:36:01.100 INFO:teuthology.orchestra.run.smithi110.stdout:55834574849 2024-05-04T17:36:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.102+0000 7f49c27fc640 1 -- 172.21.15.110:0/1646282154 >> v1:172.21.15.110:6816/783516658 conn(0x7f49bc06caf0 legacy=0x7f49bc06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.102+0000 7f49c27fc640 1 -- 172.21.15.110:0/1646282154 >> v1:172.21.15.136:6789/0 conn(0x7f49d0006a70 legacy=0x7f49d00e91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.102+0000 7f49c27fc640 1 -- 172.21.15.110:0/1646282154 shutdown_connections 2024-05-04T17:36:01.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.102+0000 7f49c27fc640 1 -- 172.21.15.110:0/1646282154 >> 172.21.15.110:0/1646282154 conn(0x7f49d00058f0 msgr2=0x7f49d00e34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.102+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb558005370 con 0x7fb57406ff00 2024-05-04T17:36:01.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.102+0000 7f49c27fc640 1 -- 172.21.15.110:0/1646282154 shutdown_connections 2024-05-04T17:36:01.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.103+0000 7f49c27fc640 1 -- 172.21.15.110:0/1646282154 wait complete. 2024-05-04T17:36:01.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.103+0000 7fb562ffd640 1 -- 172.21.15.110:0/4248935177 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 3130653186) 0x7fb5680546a0 con 0x7fb57406ff00 2024-05-04T17:36:01.107 INFO:teuthology.orchestra.run.smithi110.stdout:55834574849 2024-05-04T17:36:01.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.107+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 >> v1:172.21.15.110:6816/783516658 conn(0x7fb55406c4c0 legacy=0x7fb55406e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.107+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 >> v1:172.21.15.136:6789/0 conn(0x7fb57406ff00 legacy=0x7fb5740ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.107+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 shutdown_connections 2024-05-04T17:36:01.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.107+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 >> 172.21.15.110:0/4248935177 conn(0x7fb5740b4af0 msgr2=0x7fb574062350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.107+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 shutdown_connections 2024-05-04T17:36:01.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.107+0000 7fb578d59640 1 -- 172.21.15.110:0/4248935177 wait complete. 2024-05-04T17:36:01.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.0 2024-05-04T17:36:01.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.3 2024-05-04T17:36:01.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.111+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0c10005370 con 0x7f0c20030660 2024-05-04T17:36:01.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.111+0000 7f0c1effd640 1 -- 172.21.15.110:0/3034316308 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 326014871) 0x7f0c180537e0 con 0x7f0c20030660 2024-05-04T17:36:01.113 INFO:teuthology.orchestra.run.smithi110.stdout:60129542146 2024-05-04T17:36:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.114+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3e4c0be980 con 0x7f3e4c0c4c40 2024-05-04T17:36:01.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542146 got 60129542145 for osd.7 2024-05-04T17:36:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.114+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 >> v1:172.21.15.110:6816/783516658 conn(0x7f0c0c06d070 legacy=0x7f0c0c06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.114+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 >> v1:172.21.15.136:6789/0 conn(0x7f0c20030660 legacy=0x7f0c20169c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.114+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 shutdown_connections 2024-05-04T17:36:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.114+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 >> 172.21.15.110:0/3034316308 conn(0x7f0c20005c00 msgr2=0x7f0c200c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.115+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc90414dc00 con 0x7fc9040ae7f0 2024-05-04T17:36:01.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.114+0000 7f3e517fa640 1 -- 172.21.15.110:0/791423158 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 13373408) 0x7f3e400547f0 con 0x7f3e4c0c4c40 2024-05-04T17:36:01.117 INFO:teuthology.orchestra.run.smithi110.stdout:60129542147 2024-05-04T17:36:01.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.115+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 shutdown_connections 2024-05-04T17:36:01.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.115+0000 7f0c2f50e640 1 -- 172.21.15.110:0/3034316308 wait complete. 2024-05-04T17:36:01.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.115+0000 7fc8ebfff640 1 -- 172.21.15.110:0/434846807 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (unknown 1274345170 0 3130653186) 0x7fc8f8054cf0 con 0x7fc9040ae7f0 2024-05-04T17:36:01.118 INFO:teuthology.orchestra.run.smithi110.stdout:55834574849 2024-05-04T17:36:01.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 >> v1:172.21.15.110:6816/783516658 conn(0x7f3e3806eaa0 legacy=0x7f3e38070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 >> v1:172.21.15.136:6789/0 conn(0x7f3e4c0c4c40 legacy=0x7f3e4c0b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 shutdown_connections 2024-05-04T17:36:01.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 >> 172.21.15.110:0/791423158 conn(0x7f3e4c006530 msgr2=0x7f3e4c034cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 shutdown_connections 2024-05-04T17:36:01.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7f3e59c2a640 1 -- 172.21.15.110:0/791423158 wait complete. 2024-05-04T17:36:01.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 >> v1:172.21.15.110:6816/783516658 conn(0x7fc8f006c9e0 legacy=0x7fc8f006eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 >> v1:172.21.15.136:6789/0 conn(0x7fc9040ae7f0 legacy=0x7fc9041e36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:01.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 shutdown_connections 2024-05-04T17:36:01.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.118+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 >> 172.21.15.110:0/434846807 conn(0x7fc9040b4210 msgr2=0x7fc90406fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:01.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.119+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 shutdown_connections 2024-05-04T17:36:01.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:01.119+0000 7fc909e74640 1 -- 172.21.15.110:0/434846807 wait complete. 2024-05-04T17:36:01.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.1 2024-05-04T17:36:01.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.2 2024-05-04T17:36:01.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542146 for osd.5 2024-05-04T17:36:01.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.4 2024-05-04T17:36:01.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.6 2024-05-04T17:36:02.112 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-04T17:36:02.113 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-04T17:36:02.116 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-04T17:36:02.124 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-04T17:36:02.125 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-04T17:36:02.132 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-04T17:36:02.134 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-04T17:36:02.134 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-04T17:36:02.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.267+0000 7f7b8a704640 1 Processor -- start 2024-05-04T17:36:02.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.268+0000 7f7b8a704640 1 -- start start 2024-05-04T17:36:02.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b8a704640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b840ab350 con 0x7f7b84145f20 2024-05-04T17:36:02.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b8a704640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b840a98d0 con 0x7f7b840b4b10 2024-05-04T17:36:02.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b8a704640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b840b2410 con 0x7f7b840b4150 2024-05-04T17:36:02.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b88ca1640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7b84145f20 0x7f7b8414dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36302/0 (socket says 172.21.15.110:36302) 2024-05-04T17:36:02.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b88ca1640 1 -- 172.21.15.110:0/3360764305 learned_addr learned my addr 172.21.15.110:0/3360764305 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777496275 0 0) 0x7f7b840a98d0 con 0x7f7b840b4b10 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b6c003520 con 0x7f7b840b4b10 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.269+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886077369 0 0) 0x7f7b840b2410 con 0x7f7b840b4150 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b6c003880 con 0x7f7b840b4150 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1852299070 0 0) 0x7f7b6c003520 con 0x7f7b840b4b10 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 >> v1:172.21.15.110:6790/0 conn(0x7f7b840b4150 legacy=0x7f7b84145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 >> v1:172.21.15.110:6789/0 conn(0x7f7b84145f20 legacy=0x7f7b8414dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b840a9df0 con 0x7f7b840b4b10 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7b740015f0 con 0x7f7b840b4b10 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b740018d0 con 0x7f7b840b4b10 2024-05-04T17:36:02.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.270+0000 7f7b82ffd640 1 -- 172.21.15.110:0/3360764305 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7b74001ca0 con 0x7f7b840b4b10 2024-05-04T17:36:02.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.271+0000 7f7b8a704640 1 -- 172.21.15.110:0/3360764305 >> v1:172.21.15.136:6789/0 conn(0x7f7b840b4b10 legacy=0x7f7b8413f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.276+0000 7f7b8a704640 1 -- 172.21.15.110:0/3360764305 shutdown_connections 2024-05-04T17:36:02.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.276+0000 7f7b8a704640 1 -- 172.21.15.110:0/3360764305 >> 172.21.15.110:0/3360764305 conn(0x7f7b84061c70 msgr2=0x7f7b84062050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.276+0000 7f7b8a704640 1 -- 172.21.15.110:0/3360764305 shutdown_connections 2024-05-04T17:36:02.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.277+0000 7f7b8a704640 1 -- 172.21.15.110:0/3360764305 wait complete. 2024-05-04T17:36:02.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.278+0000 7f7b8a704640 1 Processor -- start 2024-05-04T17:36:02.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.279+0000 7f7b8a704640 1 -- start start 2024-05-04T17:36:02.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.277+0000 7fdefab6c640 1 Processor -- start 2024-05-04T17:36:02.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.278+0000 7fdefab6c640 1 -- start start 2024-05-04T17:36:02.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.279+0000 7fdefab6c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec02c440 con 0x7fdeec034850 2024-05-04T17:36:02.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7f7b83fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f7b840b4150 0x7f7b8406d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46618/0 (socket says 172.21.15.110:46618) 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.279+0000 7fdefab6c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec02aed0 con 0x7fdeec0c9550 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.279+0000 7fdefab6c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec02b190 con 0x7fdeec02aac0 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.279+0000 7fdef9b6a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdeec034850 0x7fdeec0c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36314/0 (socket says 172.21.15.110:36314) 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.279+0000 7fdef9b6a640 1 -- 172.21.15.110:0/1465345176 learned_addr learned my addr 172.21.15.110:0/1465345176 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 857677674 0 0) 0x7fdeec02c440 con 0x7fdeec034850 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdee0003520 con 0x7fdeec034850 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2762991721 0 0) 0x7fdeec02aed0 con 0x7fdeec0c9550 2024-05-04T17:36:02.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeec02c440 con 0x7fdeec0c9550 2024-05-04T17:36:02.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7f7b83fff640 1 -- 172.21.15.110:0/2602438079 learned_addr learned my addr 172.21.15.110:0/2602438079 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1523793684 0 0) 0x7fdeec02c440 con 0x7fdeec0c9550 2024-05-04T17:36:02.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 >> v1:172.21.15.110:6790/0 conn(0x7fdeec02aac0 legacy=0x7fdeec0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.280+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 >> v1:172.21.15.110:6789/0 conn(0x7fdeec034850 legacy=0x7fdeec0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec033060 con 0x7fdeec0c9550 2024-05-04T17:36:02.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8406f8d0 con 0x7f7b84145f20 2024-05-04T17:36:02.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdef40504e0 con 0x7fdeec0c9550 2024-05-04T17:36:02.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625255997 0 0) 0x7f7b840ab350 con 0x7f7b84145f20 2024-05-04T17:36:02.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8406ddc0 con 0x7f7b840b4b10 2024-05-04T17:36:02.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8406daa0 con 0x7f7b840b4150 2024-05-04T17:36:02.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.281+0000 7fdef8b68640 1 -- 172.21.15.110:0/1465345176 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdef404fb10 con 0x7fdeec0c9550 2024-05-04T17:36:02.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7fdefab6c640 1 -- 172.21.15.110:0/1465345176 >> v1:172.21.15.136:6789/0 conn(0x7fdeec0c9550 legacy=0x7fdeec0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b70003520 con 0x7f7b84145f20 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799756946 0 0) 0x7f7b8406ddc0 con 0x7f7b840b4b10 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b840ab350 con 0x7f7b840b4b10 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1890299860 0 0) 0x7f7b8406daa0 con 0x7f7b840b4150 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b8406ddc0 con 0x7f7b840b4150 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2191122251 0 0) 0x7f7b840ab350 con 0x7f7b840b4b10 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b8406daa0 con 0x7f7b840b4b10 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7b78001830 con 0x7f7b840b4b10 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 226432656 0 0) 0x7f7b70003520 con 0x7f7b84145f20 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b840ab350 con 0x7f7b84145f20 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.282+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7b7c001ad0 con 0x7f7b84145f20 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3889414401 0 0) 0x7f7b8406ddc0 con 0x7f7b840b4150 2024-05-04T17:36:02.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b70003520 con 0x7f7b840b4150 2024-05-04T17:36:02.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7fdefab6c640 1 -- 172.21.15.110:0/1465345176 shutdown_connections 2024-05-04T17:36:02.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7fdefab6c640 1 -- 172.21.15.110:0/1465345176 >> 172.21.15.110:0/1465345176 conn(0x7fdeec0058f0 msgr2=0x7fdeec005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2216603357 0 0) 0x7f7b8406daa0 con 0x7f7b840b4b10 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 >> v1:172.21.15.110:6790/0 conn(0x7f7b840b4150 legacy=0x7f7b8406d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 >> v1:172.21.15.110:6789/0 conn(0x7f7b84145f20 legacy=0x7f7b841e83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b84145350 con 0x7f7b840b4b10 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b84145640 con 0x7f7b840b4b10 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b841f03e0 con 0x7f7b840b4b10 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b780035f0 con 0x7f7b840b4b10 2024-05-04T17:36:02.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.283+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7b78003940 con 0x7f7b840b4b10 2024-05-04T17:36:02.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.285+0000 7fdefab6c640 1 -- 172.21.15.110:0/1465345176 shutdown_connections 2024-05-04T17:36:02.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.285+0000 7fdefab6c640 1 -- 172.21.15.110:0/1465345176 wait complete. 2024-05-04T17:36:02.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.286+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b64005080 con 0x7f7b840b4b10 2024-05-04T17:36:02.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.286+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f7b78019c50 con 0x7f7b840b4b10 2024-05-04T17:36:02.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.287+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f7b78085040 con 0x7f7b840b4b10 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.287+0000 7fdefab6c640 1 Processor -- start 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.287+0000 7fdefab6c640 1 -- start start 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.288+0000 7fdefab6c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec1588d0 con 0x7fdeec02aac0 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.288+0000 7fdef9b6a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdeec02aac0 0x7fdeec1581c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36334/0 (socket says 172.21.15.110:36334) 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.288+0000 7fdef9b6a640 1 -- 172.21.15.110:0/702671749 learned_addr learned my addr 172.21.15.110:0/702671749 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.289+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893382056 0 0) 0x7fdeec1588d0 con 0x7fdeec02aac0 2024-05-04T17:36:02.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.289+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec0bd470 con 0x7fdeec034850 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.289+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec0bd640 con 0x7fdeec0c9550 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.289+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fded8003520 con 0x7fdeec02aac0 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.289+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266613917 0 0) 0x7fdeec0bd640 con 0x7fdeec0c9550 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.289+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeec1588d0 con 0x7fdeec0c9550 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2420239524 0 0) 0x7fded8003520 con 0x7fdeec02aac0 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdeec0bd640 con 0x7fdeec02aac0 2024-05-04T17:36:02.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdef00018f0 con 0x7fdeec02aac0 2024-05-04T17:36:02.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1257416373 0 0) 0x7fdeec0bd640 con 0x7fdeec02aac0 2024-05-04T17:36:02.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 >> v1:172.21.15.110:6790/0 conn(0x7fdeec0c9550 legacy=0x7fdeec163ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 >> v1:172.21.15.136:6789/0 conn(0x7fdeec034850 legacy=0x7fdeec0b9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.290+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec0b9d80 con 0x7fdeec02aac0 2024-05-04T17:36:02.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.291+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdeec0ba010 con 0x7fdeec02aac0 2024-05-04T17:36:02.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.291+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdeec0bef20 con 0x7fdeec02aac0 2024-05-04T17:36:02.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.292+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdef0015880 con 0x7fdeec02aac0 2024-05-04T17:36:02.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.292+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdef0015bd0 con 0x7fdeec02aac0 2024-05-04T17:36:02.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.294+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fdef002bee0 con 0x7fdeec02aac0 2024-05-04T17:36:02.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.295+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fdef0097320 con 0x7fdeec02aac0 2024-05-04T17:36:02.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.292+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f7b78053ed0 con 0x7f7b840b4b10 2024-05-04T17:36:02.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.296+0000 7fdedffff640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded4005080 con 0x7fdeec02aac0 2024-05-04T17:36:02.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.306+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fdef00661b0 con 0x7fdeec02aac0 2024-05-04T17:36:02.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.324+0000 7fa987fff640 1 Processor -- start 2024-05-04T17:36:02.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.324+0000 7fa987fff640 1 -- start start 2024-05-04T17:36:02.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.325+0000 7fc78d02e640 1 Processor -- start 2024-05-04T17:36:02.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.325+0000 7fc78d02e640 1 -- start start 2024-05-04T17:36:02.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fc78d02e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7880aac60 con 0x7fc78813e480 2024-05-04T17:36:02.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fc78d02e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7880ab300 con 0x7fc7880ac810 2024-05-04T17:36:02.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fc78d02e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78813ef70 con 0x7fc78813bb90 2024-05-04T17:36:02.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.325+0000 7fa987fff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98002b280 con 0x7fa980029c50 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.325+0000 7fa987fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98002c3f0 con 0x7fa98002a600 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.325+0000 7fa987fff640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9800c9730 con 0x7fa9800076e0 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa9867fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa980029c50 0x7fa9800c8b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36342/0 (socket says 172.21.15.110:36342) 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa9867fc640 1 -- 172.21.15.110:0/4222453032 learned_addr learned my addr 172.21.15.110:0/4222453032 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961132018 0 0) 0x7fa98002b280 con 0x7fa980029c50 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa970003520 con 0x7fa980029c50 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721229835 0 0) 0x7fa9800c9730 con 0x7fa9800076e0 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa98002b280 con 0x7fa9800076e0 2024-05-04T17:36:02.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f733c3c7640 1 Processor -- start 2024-05-04T17:36:02.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fc78659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc7880ac810 0x7fc78813b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37310/0 (socket says 172.21.15.110:37310) 2024-05-04T17:36:02.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fc78659c640 1 -- 172.21.15.110:0/3533678449 learned_addr learned my addr 172.21.15.110:0/3533678449 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7f4f0d4e0640 1 Processor -- start 2024-05-04T17:36:02.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7f4f0d4e0640 1 -- start start 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 960748359 0 0) 0x7fa970003520 con 0x7fa980029c50 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.326+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 >> v1:172.21.15.110:6790/0 conn(0x7fa9800076e0 legacy=0x7fa9800cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 >> v1:172.21.15.136:6789/0 conn(0x7fa98002a600 legacy=0x7fa9800d7510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98002a0e0 con 0x7fa980029c50 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa97c001610 con 0x7fa980029c50 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa97c001870 con 0x7fa980029c50 2024-05-04T17:36:02.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fa985ffb640 1 -- 172.21.15.110:0/4222453032 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa97c001c40 con 0x7fa980029c50 2024-05-04T17:36:02.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f4f0d4e0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f0808bda0 con 0x7f4f0806ed40 2024-05-04T17:36:02.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f4f0d4e0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f08070760 con 0x7f4f08153a40 2024-05-04T17:36:02.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f4f0d4e0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f08070a40 con 0x7f4f080ade80 2024-05-04T17:36:02.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f4f06ffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f4f080ade80 0x7f4f080b4240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46650/0 (socket says 172.21.15.110:46650) 2024-05-04T17:36:02.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f4f06ffd640 1 -- 172.21.15.110:0/1868415029 learned_addr learned my addr 172.21.15.110:0/1868415029 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226374203 0 0) 0x7f4f08070760 con 0x7f4f08153a40 2024-05-04T17:36:02.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fa987fff640 1 -- 172.21.15.110:0/4222453032 >> v1:172.21.15.110:6789/0 conn(0x7fa980029c50 legacy=0x7fa9800c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef0003520 con 0x7f4f08153a40 2024-05-04T17:36:02.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608497328 0 0) 0x7f4f08070a40 con 0x7f4f080ade80 2024-05-04T17:36:02.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7f733c3c7640 1 -- start start 2024-05-04T17:36:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f08070760 con 0x7f4f080ade80 2024-05-04T17:36:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 634515597 0 0) 0x7f4f0808bda0 con 0x7f4f0806ed40 2024-05-04T17:36:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f08070a40 con 0x7f4f0806ed40 2024-05-04T17:36:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1413099099 0 0) 0x7f4f08070a40 con 0x7f4f0806ed40 2024-05-04T17:36:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 >> v1:172.21.15.110:6790/0 conn(0x7f4f080ade80 legacy=0x7f4f080b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 >> v1:172.21.15.136:6789/0 conn(0x7f4f08153a40 legacy=0x7f4f08153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f0806f8d0 con 0x7f4f0806ed40 2024-05-04T17:36:02.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4efc001530 con 0x7f4f0806ed40 2024-05-04T17:36:02.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fe9a1238640 1 Processor -- start 2024-05-04T17:36:02.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fe9a1238640 1 -- start start 2024-05-04T17:36:02.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110465059 0 0) 0x7fc7880aac60 con 0x7fc78813e480 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc770003520 con 0x7fc78813e480 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3646866932 0 0) 0x7fc78813ef70 con 0x7fc78813bb90 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.327+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7880aac60 con 0x7fc78813bb90 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2177469625 0 0) 0x7fc7880aac60 con 0x7fc78813bb90 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 >> v1:172.21.15.136:6789/0 conn(0x7fc7880ac810 legacy=0x7fc78813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 >> v1:172.21.15.110:6789/0 conn(0x7fc78813e480 legacy=0x7fc78813e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4efc001790 con 0x7f4f0806ed40 2024-05-04T17:36:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7f4f05ffb640 1 -- 172.21.15.110:0/1868415029 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4efc001b20 con 0x7f4f0806ed40 2024-05-04T17:36:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7880adfe0 con 0x7fc78813bb90 2024-05-04T17:36:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc77c001520 con 0x7fc78813bb90 2024-05-04T17:36:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc77c001780 con 0x7fc78813bb90 2024-05-04T17:36:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.328+0000 7fc785d9b640 1 -- 172.21.15.110:0/3533678449 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc77c001bd0 con 0x7fc78813bb90 2024-05-04T17:36:02.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fa987fff640 1 -- 172.21.15.110:0/4222453032 shutdown_connections 2024-05-04T17:36:02.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fa987fff640 1 -- 172.21.15.110:0/4222453032 >> 172.21.15.110:0/4222453032 conn(0x7fa980006530 msgr2=0x7fa980006940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fa987fff640 1 -- 172.21.15.110:0/4222453032 shutdown_connections 2024-05-04T17:36:02.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fa987fff640 1 -- 172.21.15.110:0/4222453032 wait complete. 2024-05-04T17:36:02.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fe9a1238640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99402e330 con 0x7fe99402c7d0 2024-05-04T17:36:02.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1868415029 >> v1:172.21.15.110:6789/0 conn(0x7f4f0806ed40 legacy=0x7f4f0806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fe9a1238640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99402cdd0 con 0x7fe994036a00 2024-05-04T17:36:02.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7f733c3c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7334071b80 con 0x7f73340748f0 2024-05-04T17:36:02.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7f733c3c7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7334073750 con 0x7f733413bf10 2024-05-04T17:36:02.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7f733c3c7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7334074cd0 con 0x7f733406fcd0 2024-05-04T17:36:02.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f733a964640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f733406fcd0 0x7f7334141a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46658/0 (socket says 172.21.15.110:46658) 2024-05-04T17:36:02.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f733a964640 1 -- 172.21.15.110:0/3589942346 learned_addr learned my addr 172.21.15.110:0/3589942346 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.329+0000 7fe9a1238640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940c5d80 con 0x7fe99402d2f0 2024-05-04T17:36:02.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fc78d02e640 1 -- 172.21.15.110:0/3533678449 >> v1:172.21.15.110:6790/0 conn(0x7fc78813bb90 legacy=0x7fc788149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fe99bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe994036a00 0x7fe9940c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37334/0 (socket says 172.21.15.110:37334) 2024-05-04T17:36:02.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fe99bfff640 1 -- 172.21.15.110:0/441471198 learned_addr learned my addr 172.21.15.110:0/441471198 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fc78d02e640 1 -- 172.21.15.110:0/3533678449 shutdown_connections 2024-05-04T17:36:02.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fc78d02e640 1 -- 172.21.15.110:0/3533678449 >> 172.21.15.110:0/3533678449 conn(0x7fc7880b3c50 msgr2=0x7fc7880b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fc78d02e640 1 -- 172.21.15.110:0/3533678449 shutdown_connections 2024-05-04T17:36:02.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7fc78d02e640 1 -- 172.21.15.110:0/3533678449 wait complete. 2024-05-04T17:36:02.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f341ae1a640 1 Processor -- start 2024-05-04T17:36:02.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1844675563 0 0) 0x7f7334071b80 con 0x7f73340748f0 2024-05-04T17:36:02.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7320003520 con 0x7f73340748f0 2024-05-04T17:36:02.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4092144120 0 0) 0x7f7334074cd0 con 0x7f733406fcd0 2024-05-04T17:36:02.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.330+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7334071b80 con 0x7f733406fcd0 2024-05-04T17:36:02.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 391370610 0 0) 0x7fe99402cdd0 con 0x7fe994036a00 2024-05-04T17:36:02.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe98c003520 con 0x7fe994036a00 2024-05-04T17:36:02.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1868415029 shutdown_connections 2024-05-04T17:36:02.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1868415029 >> 172.21.15.110:0/1868415029 conn(0x7f4f080623b0 msgr2=0x7f4f080b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fa987fff640 1 Processor -- start 2024-05-04T17:36:02.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fa987fff640 1 -- start start 2024-05-04T17:36:02.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1868415029 shutdown_connections 2024-05-04T17:36:02.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3917901402 0 0) 0x7fe98c003520 con 0x7fe994036a00 2024-05-04T17:36:02.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7f341ae1a640 1 -- start start 2024-05-04T17:36:02.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 >> v1:172.21.15.110:6790/0 conn(0x7fe99402d2f0 legacy=0x7fe9940c2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 >> v1:172.21.15.110:6789/0 conn(0x7fe99402c7d0 legacy=0x7fe9940d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.331+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe994009640 con 0x7fe994036a00 2024-05-04T17:36:02.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f3418bb6640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3414070850 0x7f3414070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36386/0 (socket says 172.21.15.110:36386) 2024-05-04T17:36:02.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f3418bb6640 1 -- 172.21.15.110:0/3620673017 learned_addr learned my addr 172.21.15.110:0/3620673017 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1868415029 wait complete. 2024-05-04T17:36:02.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7fa9867fc640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fa98002a600 0x7fa9800cab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46672/0 (socket says 172.21.15.110:46672) 2024-05-04T17:36:02.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7fa9867fc640 1 -- 172.21.15.110:0/780986364 learned_addr learned my addr 172.21.15.110:0/780986364 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1101834481 0 0) 0x7f7320003520 con 0x7f73340748f0 2024-05-04T17:36:02.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 >> v1:172.21.15.110:6790/0 conn(0x7f733406fcd0 legacy=0x7f7334141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 >> v1:172.21.15.136:6789/0 conn(0x7f733413bf10 legacy=0x7f733406f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7334074ea0 con 0x7f73340748f0 2024-05-04T17:36:02.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f732c001610 con 0x7f73340748f0 2024-05-04T17:36:02.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe9880015f0 con 0x7fe994036a00 2024-05-04T17:36:02.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3414072830 con 0x7f3414070850 2024-05-04T17:36:02.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3414072aa0 con 0x7f34140b0600 2024-05-04T17:36:02.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f732c001870 con 0x7f73340748f0 2024-05-04T17:36:02.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f7339161640 1 -- 172.21.15.110:0/3589942346 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f732c001c40 con 0x7f73340748f0 2024-05-04T17:36:02.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3414140540 con 0x7f34140afcb0 2024-05-04T17:36:02.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7fc78d02e640 1 Processor -- start 2024-05-04T17:36:02.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024519072 0 0) 0x7f3408001110 con 0x7f34140b0600 2024-05-04T17:36:02.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98002f040 con 0x7fa9800cb250 2024-05-04T17:36:02.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33fc003520 con 0x7f34140b0600 2024-05-04T17:36:02.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4031923309 0 0) 0x7fa98002f040 con 0x7fa9800cb250 2024-05-04T17:36:02.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3461100119 0 0) 0x7f33fc003520 con 0x7f34140b0600 2024-05-04T17:36:02.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98002f2b0 con 0x7fa9800076e0 2024-05-04T17:36:02.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 >> v1:172.21.15.110:6790/0 conn(0x7f34140afcb0 legacy=0x7f341414b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98002f590 con 0x7fa98002a600 2024-05-04T17:36:02.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 >> v1:172.21.15.110:6789/0 conn(0x7f3414070850 legacy=0x7f3414070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3414072d80 con 0x7f34140b0600 2024-05-04T17:36:02.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f34080017e0 con 0x7f34140b0600 2024-05-04T17:36:02.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa974003520 con 0x7fa9800cb250 2024-05-04T17:36:02.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3408001a40 con 0x7f34140b0600 2024-05-04T17:36:02.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f34137fe640 1 -- 172.21.15.110:0/3620673017 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f34080035f0 con 0x7f34140b0600 2024-05-04T17:36:02.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.332+0000 7fc78d02e640 1 -- start start 2024-05-04T17:36:02.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f733c3c7640 1 -- 172.21.15.110:0/3589942346 >> v1:172.21.15.110:6789/0 conn(0x7f73340748f0 legacy=0x7f733413b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 667009824 0 0) 0x7fa974003520 con 0x7fa9800cb250 2024-05-04T17:36:02.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa98002f040 con 0x7fa9800cb250 2024-05-04T17:36:02.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fc78d02e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78813f5c0 con 0x7fc78813bb90 2024-05-04T17:36:02.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fc78d02e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc788141f90 con 0x7fc78813e480 2024-05-04T17:36:02.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fc78d02e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc788142160 con 0x7fc7880ac810 2024-05-04T17:36:02.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f733c3c7640 1 -- 172.21.15.110:0/3589942346 shutdown_connections 2024-05-04T17:36:02.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7f733c3c7640 1 -- 172.21.15.110:0/3589942346 >> 172.21.15.110:0/3589942346 conn(0x7f73340b4510 msgr2=0x7f73340b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa98806a050 con 0x7fa9800cb250 2024-05-04T17:36:02.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fc78759e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc78813e480 0x7fc7881dbf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37366/0 (socket says 172.21.15.110:37366) 2024-05-04T17:36:02.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fc78759e640 1 -- 172.21.15.110:0/3176299366 learned_addr learned my addr 172.21.15.110:0/3176299366 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7f733c3c7640 1 -- 172.21.15.110:0/3589942346 shutdown_connections 2024-05-04T17:36:02.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.333+0000 7fe99affd640 1 -- 172.21.15.110:0/441471198 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe988001920 con 0x7fe994036a00 2024-05-04T17:36:02.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3400345472 0 0) 0x7fc788141f90 con 0x7fc78813e480 2024-05-04T17:36:02.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc774003520 con 0x7fc78813e480 2024-05-04T17:36:02.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422531453 0 0) 0x7fa98002f590 con 0x7fa98002a600 2024-05-04T17:36:02.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa974003520 con 0x7fa98002a600 2024-05-04T17:36:02.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7f733c3c7640 1 -- 172.21.15.110:0/3589942346 wait complete. 2024-05-04T17:36:02.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 >> v1:172.21.15.136:6789/0 conn(0x7f34140b0600 legacy=0x7f3414140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 shutdown_connections 2024-05-04T17:36:02.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 >> 172.21.15.110:0/3620673017 conn(0x7f34140b5c00 msgr2=0x7f34140b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093635280 0 0) 0x7fc78813f5c0 con 0x7fc78813bb90 2024-05-04T17:36:02.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc788141f90 con 0x7fc78813bb90 2024-05-04T17:36:02.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.334+0000 7fe9a1238640 1 -- 172.21.15.110:0/441471198 >> v1:172.21.15.136:6789/0 conn(0x7fe994036a00 legacy=0x7fe9940c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2041849114 0 0) 0x7fa98002f040 con 0x7fa9800cb250 2024-05-04T17:36:02.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fe9a1238640 1 -- 172.21.15.110:0/441471198 shutdown_connections 2024-05-04T17:36:02.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fe9a1238640 1 -- 172.21.15.110:0/441471198 >> 172.21.15.110:0/441471198 conn(0x7fe994005c60 msgr2=0x7fe994006070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 >> v1:172.21.15.110:6790/0 conn(0x7fa98002a600 legacy=0x7fa9800cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 >> v1:172.21.15.136:6789/0 conn(0x7fa9800076e0 legacy=0x7fa98002d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98002de10 con 0x7fa9800cb250 2024-05-04T17:36:02.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7f4f0d4e0640 1 Processor -- start 2024-05-04T17:36:02.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9880504e0 con 0x7fa9800cb250 2024-05-04T17:36:02.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa98804fe40 con 0x7fa9800cb250 2024-05-04T17:36:02.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4183695720 0 0) 0x7fc788141f90 con 0x7fc78813bb90 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc78813f5c0 con 0x7fc78813bb90 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc778001720 con 0x7fc78813bb90 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1832890587 0 0) 0x7fc7800015b0 con 0x7fc7880ac810 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc788141f90 con 0x7fc7880ac810 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2752368467 0 0) 0x7fc774003520 con 0x7fc78813e480 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7800015b0 con 0x7fc78813e480 2024-05-04T17:36:02.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc77c001b10 con 0x7fc78813e480 2024-05-04T17:36:02.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 shutdown_connections 2024-05-04T17:36:02.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7f733c3c7640 1 Processor -- start 2024-05-04T17:36:02.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7f733c3c7640 1 -- start start 2024-05-04T17:36:02.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7f4f0d4e0640 1 -- start start 2024-05-04T17:36:02.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.336+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3860206493 0 0) 0x7fc788141f90 con 0x7fc7880ac810 2024-05-04T17:36:02.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.336+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc774003520 con 0x7fc7880ac810 2024-05-04T17:36:02.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.336+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc780001e40 con 0x7fc7880ac810 2024-05-04T17:36:02.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 449375692 0 0) 0x7fc774003520 con 0x7fc7880ac810 2024-05-04T17:36:02.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f733c3c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f733412ad70 con 0x7f733413bf10 2024-05-04T17:36:02.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.336+0000 7fe9a1238640 1 -- 172.21.15.110:0/441471198 shutdown_connections 2024-05-04T17:36:02.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.136:6789/0 conn(0x7fc78813e480 legacy=0x7fc7881dbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f733c3c7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f733413e6c0 con 0x7f733406fcd0 2024-05-04T17:36:02.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f733c3c7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f733413dc10 con 0x7f73340748f0 2024-05-04T17:36:02.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6789/0 conn(0x7fc78813bb90 legacy=0x7fc7881db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f733a163640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f733406fcd0 0x7f733413c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37378/0 (socket says 172.21.15.110:37378) 2024-05-04T17:36:02.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f733a163640 1 -- 172.21.15.110:0/1978344186 learned_addr learned my addr 172.21.15.110:0/1978344186 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc788140740 con 0x7fc7880ac810 2024-05-04T17:36:02.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f4f06ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4f0806ed40 0x7f4f080885f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.110:37376/0 (socket says 172.21.15.110:37376) 2024-05-04T17:36:02.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f4f067fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4f080ade80 0x7f4f080ab6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36416/0 (socket says 172.21.15.110:36416) 2024-05-04T17:36:02.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249852474 0 0) 0x7f733413e6c0 con 0x7f733406fcd0 2024-05-04T17:36:02.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc780003c20 con 0x7fc7880ac810 2024-05-04T17:36:02.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc780003f70 con 0x7fc7880ac810 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.336+0000 7f341ae1a640 1 -- 172.21.15.110:0/3620673017 wait complete. 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7318003520 con 0x7f733406fcd0 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1576343641 0 0) 0x7f733413dc10 con 0x7f73340748f0 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73180048e0 con 0x7f73340748f0 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2622830157 0 0) 0x7f73180048e0 con 0x7f73340748f0 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318004d90 con 0x7f73340748f0 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7330001c30 con 0x7f73340748f0 2024-05-04T17:36:02.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 898840859 0 0) 0x7f7318004d90 con 0x7f73340748f0 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 >> v1:172.21.15.136:6789/0 conn(0x7f733406fcd0 legacy=0x7f733413c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 >> v1:172.21.15.110:6789/0 conn(0x7f733413bf10 legacy=0x7f733413ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7334129da0 con 0x7f73340748f0 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f733412c580 con 0x7f73340748f0 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f733412ca90 con 0x7f73340748f0 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7330003970 con 0x7f73340748f0 2024-05-04T17:36:02.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7330001a90 con 0x7f73340748f0 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fe9a1238640 1 -- 172.21.15.110:0/441471198 wait complete. 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7fe9a1238640 1 Processor -- start 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7fe9a1238640 1 -- start start 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fe9a1238640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940bfc50 con 0x7fe99402c7d0 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fe9a1238640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940cf360 con 0x7fe994036a00 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fe9a1238640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940beb60 con 0x7fe99402d2f0 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fe99b7fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fe99402d2f0 0x7fe9940cfa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46716/0 (socket says 172.21.15.110:46716) 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fe99b7fe640 1 -- 172.21.15.110:0/2633121617 learned_addr learned my addr 172.21.15.110:0/2633121617 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7f341ae1a640 1 Processor -- start 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7f341ae1a640 1 -- start start 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f341ae1a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34141308b0 con 0x7f34140afcb0 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f341ae1a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3414132680 con 0x7f3414070850 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f341ae1a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3414132960 con 0x7f34140b0600 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f3413fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f34140afcb0 0x7f341412fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36428/0 (socket says 172.21.15.110:36428) 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f3413fff640 1 -- 172.21.15.110:0/513395025 learned_addr learned my addr 172.21.15.110:0/513395025 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660257955 0 0) 0x7f3414132960 con 0x7f34140b0600 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.339+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3400003520 con 0x7f34140b0600 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3096811599 0 0) 0x7f3400003520 con 0x7f34140b0600 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3414132960 con 0x7f34140b0600 2024-05-04T17:36:02.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f340c001c10 con 0x7f34140b0600 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7881409d0 con 0x7fc7880ac810 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7881e7340 con 0x7fc7880ac810 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7fc786d9d640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6790/0 conn(0x7fc7880ac810 legacy=0x7fc788140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7fc786d9d640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6790/0 conn(0x7fc7880ac810 legacy=0x7fc788140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7fc786d9d640 1 --1- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6790/0 conn(0x7fc7880ac810 0x7fc788140030 :-1 s=OPENED pgs=125 cs=1 l=1).handle_message read tag failed 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7fc786d9d640 1 --1- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6790/0 conn(0x7fc7880ac810 0x7fc788140030 :-1 s=OPENED pgs=125 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.338+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6790/0 conn(0x7fc7880ac810 legacy=0x7fc788140030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc774003520 con 0x7fc78813bb90 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc788141f90 con 0x7fc78813e480 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc774012660 con 0x7fc7740051d0 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890324632 0 0) 0x7fc788141f90 con 0x7fc78813e480 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fc774012830 con 0x7fc78813e480 2024-05-04T17:36:02.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3374118258 0 0) 0x7fc774012830 con 0x7fc78813e480 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6790/0 conn(0x7fc7740051d0 legacy=0x7fc77400b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6789/0 conn(0x7fc78813bb90 legacy=0x7fc77400eaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fc7740139e0 con 0x7fc78813e480 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc78804fe40 con 0x7fc78813e480 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc774003520 con 0x7fc78813e480 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fc780005570 con 0x7fc78813e480 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7f4f06ffd640 1 -- 172.21.15.110:0/1390511815 learned_addr learned my addr 172.21.15.110:0/1390511815 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f0814c570 con 0x7f4f080ade80 2024-05-04T17:36:02.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f0814c850 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f0814cb30 con 0x7f4f08153a40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4231596529 0 0) 0x7f4f0814c850 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef4003520 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 320685626 0 0) 0x7f4ef4003520 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f0814c850 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4ef8001830 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1866856158 0 0) 0x7f4f0814c850 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 >> v1:172.21.15.110:6790/0 conn(0x7f4f08153a40 legacy=0x7f4f081f2f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 >> v1:172.21.15.110:6789/0 conn(0x7f4f080ade80 legacy=0x7f4f080ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f0814aea0 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f0814b7a0 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f080ad1d0 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ef80047e0 con 0x7f4f0806ed40 2024-05-04T17:36:02.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4ef8004b30 con 0x7f4f0806ed40 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa98002daa0 con 0x7fa9800cb250 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.335+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9800c3750 con 0x7fa9800cb250 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.337+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa980005120 con 0x7fa9800cb250 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.340+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fa988068240 con 0x7fa9800cb250 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.341+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fa9880ffa70 con 0x7fa9800cb250 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2111860819 0 0) 0x7fe9940cf360 con 0x7fe994036a00 2024-05-04T17:36:02.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.342+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe980003520 con 0x7fe994036a00 2024-05-04T17:36:02.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 829849220 0 0) 0x7f3414132960 con 0x7f34140b0600 2024-05-04T17:36:02.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 371621665 0 0) 0x7fe9940beb60 con 0x7fe99402d2f0 2024-05-04T17:36:02.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9940cf360 con 0x7fe99402d2f0 2024-05-04T17:36:02.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 >> v1:172.21.15.136:6789/0 conn(0x7f3414070850 legacy=0x7f34141d6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 >> v1:172.21.15.110:6789/0 conn(0x7f34140afcb0 legacy=0x7f341412fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3414132c40 con 0x7f34140b0600 2024-05-04T17:36:02.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f7330019d10 con 0x7f73340748f0 2024-05-04T17:36:02.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f7330084cb0 con 0x7f73340748f0 2024-05-04T17:36:02.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.343+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc780005440 con 0x7fc78813e480 2024-05-04T17:36:02.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.344+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2074978015 0 0) 0x7fe9940cf360 con 0x7fe99402d2f0 2024-05-04T17:36:02.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.344+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9940beb60 con 0x7fe99402d2f0 2024-05-04T17:36:02.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.344+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe990001950 con 0x7fe99402d2f0 2024-05-04T17:36:02.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.344+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3414138e20 con 0x7f34140b0600 2024-05-04T17:36:02.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.344+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f34141d7720 con 0x7f34140b0600 2024-05-04T17:36:02.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f731c005080 con 0x7f73340748f0 2024-05-04T17:36:02.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f340c002010 con 0x7f34140b0600 2024-05-04T17:36:02.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f340c003de0 con 0x7f34140b0600 2024-05-04T17:36:02.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f4ef8004cd0 con 0x7f4f0806ed40 2024-05-04T17:36:02.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fc78001b900 con 0x7fc78813e480 2024-05-04T17:36:02.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f4ef80861e0 con 0x7f4f0806ed40 2024-05-04T17:36:02.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.345+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ee8005080 con 0x7f4f0806ed40 2024-05-04T17:36:02.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 6 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fc780086b00 con 0x7fc78813e480 2024-05-04T17:36:02.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2002215927 0 0) 0x7fe9940beb60 con 0x7fe99402d2f0 2024-05-04T17:36:02.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 >> v1:172.21.15.136:6789/0 conn(0x7fe994036a00 legacy=0x7fe99416bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 >> v1:172.21.15.110:6789/0 conn(0x7fe99402c7d0 legacy=0x7fe9940c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9940d0170 con 0x7fe99402d2f0 2024-05-04T17:36:02.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe9940bdfb0 con 0x7fe99402d2f0 2024-05-04T17:36:02.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9940bf290 con 0x7fe99402d2f0 2024-05-04T17:36:02.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.346+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f4005080 con 0x7f34140b0600 2024-05-04T17:36:02.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.347+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fa9880ce2b0 con 0x7fa9800cb250 2024-05-04T17:36:02.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.347+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fc788142160 con 0x7fc78813e480 2024-05-04T17:36:02.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.348+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9940be220 con 0x7fe99402d2f0 2024-05-04T17:36:02.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.352+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fc780055990 con 0x7fc78813e480 2024-05-04T17:36:02.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.356+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f7330053c80 con 0x7f73340748f0 2024-05-04T17:36:02.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.356+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe990001cd0 con 0x7fe99402d2f0 2024-05-04T17:36:02.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.357+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f340c01a0f0 con 0x7f34140b0600 2024-05-04T17:36:02.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.358+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f340c085370 con 0x7f34140b0600 2024-05-04T17:36:02.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.358+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fe990003a60 con 0x7fe99402d2f0 2024-05-04T17:36:02.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.358+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fe990003c40 con 0x7fe99402d2f0 2024-05-04T17:36:02.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.359+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f4ef8055070 con 0x7f4f0806ed40 2024-05-04T17:36:02.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.367+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f340c054200 con 0x7f34140b0600 2024-05-04T17:36:02.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.367+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7fe990052e30 con 0x7fe99402d2f0 2024-05-04T17:36:02.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.376+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fe990055230 con 0x7fe99402d2f0 2024-05-04T17:36:02.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.698+0000 7f7b8a704640 1 -- 172.21.15.110:0/2602438079 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7b64005370 con 0x7f7b840b4b10 2024-05-04T17:36:02.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.698+0000 7f7b80ff9640 1 -- 172.21.15.110:0/2602438079 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 2194451396) 0x7f7b780537d0 con 0x7f7b840b4b10 2024-05-04T17:36:02.700 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2024-05-04T17:36:02.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.702+0000 7f7b6a7fc640 1 -- 172.21.15.110:0/2602438079 >> v1:172.21.15.110:6816/783516658 conn(0x7f7b7006cbf0 legacy=0x7f7b7006f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.702+0000 7f7b6a7fc640 1 -- 172.21.15.110:0/2602438079 >> v1:172.21.15.136:6789/0 conn(0x7f7b840b4b10 legacy=0x7f7b841e7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.702+0000 7f7b6a7fc640 1 -- 172.21.15.110:0/2602438079 shutdown_connections 2024-05-04T17:36:02.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.702+0000 7f7b6a7fc640 1 -- 172.21.15.110:0/2602438079 >> 172.21.15.110:0/2602438079 conn(0x7f7b84061c70 msgr2=0x7f7b840b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.703+0000 7f7b6a7fc640 1 -- 172.21.15.110:0/2602438079 shutdown_connections 2024-05-04T17:36:02.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.703+0000 7f7b6a7fc640 1 -- 172.21.15.110:0/2602438079 wait complete. 2024-05-04T17:36:02.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.708+0000 7fdedffff640 1 -- 172.21.15.110:0/702671749 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fded4005370 con 0x7fdeec02aac0 2024-05-04T17:36:02.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.709+0000 7fdeea7fc640 1 -- 172.21.15.110:0/702671749 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 2194451396) 0x7fdef0065ab0 con 0x7fdeec02aac0 2024-05-04T17:36:02.710 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2024-05-04T17:36:02.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.713+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 >> v1:172.21.15.110:6816/783516658 conn(0x7fded806ca80 legacy=0x7fded806ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.713+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 >> v1:172.21.15.110:6789/0 conn(0x7fdeec02aac0 legacy=0x7fdeec1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.714+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 shutdown_connections 2024-05-04T17:36:02.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.714+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 >> 172.21.15.110:0/702671749 conn(0x7fdeec0058f0 msgr2=0x7fdeec0c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.714+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 shutdown_connections 2024-05-04T17:36:02.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.714+0000 7fdefab6c640 1 -- 172.21.15.110:0/702671749 wait complete. 2024-05-04T17:36:02.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2024-05-04T17:36:02.721 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2024-05-04T17:36:02.732 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.746+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4ee8005370 con 0x7f4f0806ed40 2024-05-04T17:36:02.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.747+0000 7f4eef7fe640 1 -- 172.21.15.110:0/1390511815 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 3887400221) 0x7f4ef8054970 con 0x7f4f0806ed40 2024-05-04T17:36:02.748 INFO:teuthology.orchestra.run.smithi110.stdout:60129542148 2024-05-04T17:36:02.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.749+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 >> v1:172.21.15.110:6816/783516658 conn(0x7f4ef406ca40 legacy=0x7f4ef406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.749+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 >> v1:172.21.15.136:6789/0 conn(0x7f4f0806ed40 legacy=0x7f4f080885f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.749+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 shutdown_connections 2024-05-04T17:36:02.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.749+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 >> 172.21.15.110:0/1390511815 conn(0x7f4f080623b0 msgr2=0x7f4f080b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.749+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe994005120 con 0x7fe99402d2f0 2024-05-04T17:36:02.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.750+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 shutdown_connections 2024-05-04T17:36:02.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.750+0000 7f4f0d4e0640 1 -- 172.21.15.110:0/1390511815 wait complete. 2024-05-04T17:36:02.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.750+0000 7fe998ff9640 1 -- 172.21.15.110:0/2633121617 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 2439918515) 0x7fe990054b30 con 0x7fe99402d2f0 2024-05-04T17:36:02.752 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:02.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.752+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 >> v1:172.21.15.110:6816/783516658 conn(0x7fe98006c940 legacy=0x7fe98006ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.752+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 >> v1:172.21.15.110:6790/0 conn(0x7fe99402d2f0 legacy=0x7fe9940cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.752+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 shutdown_connections 2024-05-04T17:36:02.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.752+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 >> 172.21.15.110:0/2633121617 conn(0x7fe994005c60 msgr2=0x7fe994038c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.753+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 shutdown_connections 2024-05-04T17:36:02.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.753+0000 7fe9a1238640 1 -- 172.21.15.110:0/2633121617 wait complete. 2024-05-04T17:36:02.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.4 2024-05-04T17:36:02.763 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.762+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f731c005370 con 0x7f73340748f0 2024-05-04T17:36:02.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.763+0000 7f732affd640 1 -- 172.21.15.110:0/1978344186 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (unknown 1482059429 0 326014871) 0x7f7330053c80 con 0x7f73340748f0 2024-05-04T17:36:02.764 INFO:teuthology.orchestra.run.smithi110.stdout:60129542146 2024-05-04T17:36:02.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.1 2024-05-04T17:36:02.767 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 >> v1:172.21.15.110:6816/783516658 conn(0x7f731806d500 legacy=0x7f731806f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 >> v1:172.21.15.110:6790/0 conn(0x7f73340748f0 legacy=0x7f7334140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 shutdown_connections 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 >> 172.21.15.110:0/1978344186 conn(0x7f73340b4510 msgr2=0x7f73340afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 shutdown_connections 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7f733c3c7640 1 -- 172.21.15.110:0/1978344186 wait complete. 2024-05-04T17:36:02.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa9800cfbc0 con 0x7fa9800cb250 2024-05-04T17:36:02.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.765+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc78806b1a0 con 0x7fc78813e480 2024-05-04T17:36:02.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.766+0000 7fa96f7fe640 1 -- 172.21.15.110:0/780986364 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (unknown 1274345170 0 2439918515) 0x7fa9880d25b0 con 0x7fa9800cb250 2024-05-04T17:36:02.770 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:02.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.766+0000 7fc76f7fe640 1 -- 172.21.15.110:0/3176299366 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 13373408) 0x7fc780055290 con 0x7fc78813e480 2024-05-04T17:36:02.770 INFO:teuthology.orchestra.run.smithi110.stdout:60129542147 2024-05-04T17:36:02.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 >> v1:172.21.15.110:6816/783516658 conn(0x7fa97406c7d0 legacy=0x7fa97406ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 >> v1:172.21.15.110:6789/0 conn(0x7fa9800cb250 legacy=0x7fa9800c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 shutdown_connections 2024-05-04T17:36:02.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 >> 172.21.15.110:0/780986364 conn(0x7fa980006530 msgr2=0x7fa980038270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.110:6816/783516658 conn(0x7fc7740786f0 legacy=0x7fc77407abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 >> v1:172.21.15.136:6789/0 conn(0x7fc78813e480 legacy=0x7fc7740079a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 shutdown_connections 2024-05-04T17:36:02.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fa987fff640 1 -- 172.21.15.110:0/780986364 wait complete. 2024-05-04T17:36:02.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 shutdown_connections 2024-05-04T17:36:02.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 >> 172.21.15.110:0/3176299366 conn(0x7fc7880b3c50 msgr2=0x7fc7880b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 shutdown_connections 2024-05-04T17:36:02.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.769+0000 7fc78d02e640 1 -- 172.21.15.110:0/3176299366 wait complete. 2024-05-04T17:36:02.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.773+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f33f4005370 con 0x7f34140b0600 2024-05-04T17:36:02.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.774+0000 7f34117fa640 1 -- 172.21.15.110:0/513395025 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 2439918515) 0x7f340c053b00 con 0x7f34140b0600 2024-05-04T17:36:02.775 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2024-05-04T17:36:02.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.776+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 >> v1:172.21.15.110:6816/783516658 conn(0x7f340006c940 legacy=0x7f340006ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.776+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 >> v1:172.21.15.110:6790/0 conn(0x7f34140b0600 legacy=0x7f34141301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.776+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 shutdown_connections 2024-05-04T17:36:02.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.776+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 >> 172.21.15.110:0/513395025 conn(0x7f34140b5c00 msgr2=0x7f34140b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.776+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 shutdown_connections 2024-05-04T17:36:02.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.776+0000 7f341ae1a640 1 -- 172.21.15.110:0/513395025 wait complete. 2024-05-04T17:36:02.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542146 got 60129542146 for osd.7 2024-05-04T17:36:02.780 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.6 2024-05-04T17:36:02.789 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.5 2024-05-04T17:36:02.790 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.2 2024-05-04T17:36:02.791 DEBUG:teuthology.parallel:result is None 2024-05-04T17:36:02.791 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-04T17:36:02.791 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:36:02.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.923+0000 7f0dc2606640 1 Processor -- start 2024-05-04T17:36:02.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.923+0000 7f0dc2606640 1 -- start start 2024-05-04T17:36:02.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dc2606640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc0aaa70 con 0x7f0dbc13bf60 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dc2606640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc0ae040 con 0x7f0dbc13e640 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dc2606640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc0ab5f0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dbbfff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0dbc0acfc0 0x7f0dbc0ad3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46726/0 (socket says 172.21.15.110:46726) 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dbbfff640 1 -- 172.21.15.110:0/1615089945 learned_addr learned my addr 172.21.15.110:0/1615089945 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dc0ba3640 1 --1- 172.21.15.110:0/1615089945 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13bf60 0x7f0dbc149f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36448/0 (socket says 172.21.15.110:36448) 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659118745 0 0) 0x7f0dbc0ab5f0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0da8003520 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440588809 0 0) 0x7f0dbc0aaa70 con 0x7f0dbc13bf60 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.924+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dbc0ab5f0 con 0x7f0dbc13bf60 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2668966020 0 0) 0x7f0dbc0ae040 con 0x7f0dbc13e640 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dbc0aaa70 con 0x7f0dbc13e640 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2565729448 0 0) 0x7f0da8003520 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 >> v1:172.21.15.136:6789/0 conn(0x7f0dbc13e640 legacy=0x7f0dbc13b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13bf60 legacy=0x7f0dbc149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dbc0aa690 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0da4001610 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0da40018b0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dbaffd640 1 -- 172.21.15.110:0/1615089945 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0da4001cc0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dc2606640 1 -- 172.21.15.110:0/1615089945 >> v1:172.21.15.110:6790/0 conn(0x7f0dbc0acfc0 legacy=0x7f0dbc0ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dc2606640 1 -- 172.21.15.110:0/1615089945 shutdown_connections 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dc2606640 1 -- 172.21.15.110:0/1615089945 >> 172.21.15.110:0/1615089945 conn(0x7f0dbc0b3c50 msgr2=0x7f0dbc0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dc2606640 1 -- 172.21.15.110:0/1615089945 shutdown_connections 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.925+0000 7f0dc2606640 1 -- 172.21.15.110:0/1615089945 wait complete. 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.926+0000 7f0dc2606640 1 Processor -- start 2024-05-04T17:36:02.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc2606640 1 -- start start 2024-05-04T17:36:02.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc2606640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc141f70 con 0x7f0dbc13e640 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc2606640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc1d7ee0 con 0x7f0dbc13bf60 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc2606640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc141120 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dbbfff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0dbc0acfc0 0x7f0dbc142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46742/0 (socket says 172.21.15.110:46742) 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dbbfff640 1 -- 172.21.15.110:0/3671403059 learned_addr learned my addr 172.21.15.110:0/3671403059 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc0ba3640 1 -- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13e640 legacy=0x7f0dbc1d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc0ba3640 1 -- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13e640 legacy=0x7f0dbc1d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc0ba3640 1 --1- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13e640 0x7f0dbc1d77d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0dc0ba3640 1 --1- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13e640 0x7f0dbc1d77d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2144382392 0 0) 0x7f0dbc141120 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d9c003520 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3566239068 0 0) 0x7f0d9c003520 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dbc141120 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0da4001510 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.927+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2150995538 0 0) 0x7f0dbc1d7ee0 con 0x7f0dbc13bf60 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d9c003520 con 0x7f0dbc13bf60 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1396636059 0 0) 0x7f0dbc141120 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 >> v1:172.21.15.136:6789/0 conn(0x7f0dbc13bf60 legacy=0x7f0dbc131550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6789/0 conn(0x7f0dbc13e640 legacy=0x7f0dbc1d77d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dbc1412f0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0dbc141a30 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0dbc072840 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0da4004ce0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.928+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0da4004f10 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.929+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da0005080 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.929+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f0da40050b0 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.929+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f0da4086450 con 0x7f0dbc0acfc0 2024-05-04T17:36:02.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:02.933+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f0da40552e0 con 0x7f0dbc0acfc0 2024-05-04T17:36:03.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.104+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0da0000b90 con 0x7f0d9c06ebd0 2024-05-04T17:36:03.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.105+0000 7f0db8ff9640 1 -- 172.21.15.110:0/3671403059 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35484 (unknown 2965378022 0 1030008286) 0x7f0da0000b90 con 0x7f0d9c06ebd0 2024-05-04T17:36:03.107 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:36:03.107 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:36:03.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.107+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6816/783516658 conn(0x7f0d9c06ebd0 legacy=0x7f0d9c071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.107+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 >> v1:172.21.15.110:6790/0 conn(0x7f0dbc0acfc0 legacy=0x7f0dbc142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.107+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 shutdown_connections 2024-05-04T17:36:03.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.107+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 >> 172.21.15.110:0/3671403059 conn(0x7f0dbc0b3c50 msgr2=0x7f0dbc0ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:03.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.107+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 shutdown_connections 2024-05-04T17:36:03.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.107+0000 7f0dc2606640 1 -- 172.21.15.110:0/3671403059 wait complete. 2024-05-04T17:36:03.119 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-05-04T17:36:01.124794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":223368,"kb_used_data":2144,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751352,"statfs":{"total":773094113280,"available":772865384448,"internally_reserved":0,"allocated":2195456,"data_stored":1396568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.223029+0000","last_change":"2024-05-04T17:35:59.221521+0000","last_active":"2024-05-04T17:35:59.223029+0000","last_peered":"2024-05-04T17:35:59.223029+0000","last_clean":"2024-05-04T17:35:59.223029+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:35:59.223029+0000","last_undegraded":"2024-05-04T17:35:59.223029+0000","last_fullsized":"2024-05-04T17:35:59.223029+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:27:44.959711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.222600+0000","last_change":"2024-05-04T17:35:59.221461+0000","last_active":"2024-05-04T17:35:59.222600+0000","last_peered":"2024-05-04T17:35:59.222600+0000","last_clean":"2024-05-04T17:35:59.222600+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:35:59.222600+0000","last_undegraded":"2024-05-04T17:35:59.222600+0000","last_fullsized":"2024-05-04T17:35:59.222600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:43:34.034451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.223837+0000","last_change":"2024-05-04T17:35:59.222186+0000","last_active":"2024-05-04T17:35:59.223837+0000","last_peered":"2024-05-04T17:35:59.223837+0000","last_clean":"2024-05-04T17:35:59.223837+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:35:59.223837+0000","last_undegraded":"2024-05-04T17:35:59.223837+0000","last_fullsized":"2024-05-04T17:35:59.223837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:40:36.227188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.219295+0000","last_change":"2024-05-04T17:35:59.218744+0000","last_active":"2024-05-04T17:35:59.219295+0000","last_peered":"2024-05-04T17:35:59.219295+0000","last_clean":"2024-05-04T17:35:59.219295+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:35:59.219295+0000","last_undegraded":"2024-05-04T17:35:59.219295+0000","last_fullsized":"2024-05-04T17:35:59.219295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:01:01.767123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:00.220271+0000","last_change":"2024-05-04T17:35:59.223094+0000","last_active":"2024-05-04T17:36:00.220271+0000","last_peered":"2024-05-04T17:36:00.220271+0000","last_clean":"2024-05-04T17:36:00.220271+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:36:00.220271+0000","last_undegraded":"2024-05-04T17:36:00.220271+0000","last_fullsized":"2024-05-04T17:36:00.220271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:54:23.823020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.331823+0000","last_change":"2024-05-04T17:35:59.331384+0000","last_active":"2024-05-04T17:35:59.331823+0000","last_peered":"2024-05-04T17:35:59.331823+0000","last_clean":"2024-05-04T17:35:59.331823+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:35:59.331823+0000","last_undegraded":"2024-05-04T17:35:59.331823+0000","last_fullsized":"2024-05-04T17:35:59.331823+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:01:01.847380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.224710+0000","last_change":"2024-05-04T17:35:59.223322+0000","last_active":"2024-05-04T17:35:59.224710+0000","last_peered":"2024-05-04T17:35:59.224710+0000","last_clean":"2024-05-04T17:35:59.224710+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:35:59.224710+0000","last_undegraded":"2024-05-04T17:35:59.224710+0000","last_fullsized":"2024-05-04T17:35:59.224710+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:23:10.183237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:00.220272+0000","last_change":"2024-05-04T17:35:59.221969+0000","last_active":"2024-05-04T17:36:00.220272+0000","last_peered":"2024-05-04T17:36:00.220272+0000","last_clean":"2024-05-04T17:36:00.220272+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:36:00.220272+0000","last_undegraded":"2024-05-04T17:36:00.220272+0000","last_fullsized":"2024-05-04T17:36:00.220272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:22:19.190308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.232135+0000","last_change":"2024-05-04T17:35:58.172775+0000","last_active":"2024-05-04T17:35:59.232135+0000","last_peered":"2024-05-04T17:35:59.232135+0000","last_clean":"2024-05-04T17:35:59.232135+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:35:59.232135+0000","last_undegraded":"2024-05-04T17:35:59.232135+0000","last_fullsized":"2024-05-04T17:35:59.232135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_clean_scrub_stamp":"2024-05-04T17:35:57.156954+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:43:38.793258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542146,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":671744,"data_stored":570110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":671744,"data_stored":570110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:36:03.120 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:36:03.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f098128c640 1 Processor -- start 2024-05-04T17:36:03.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f098128c640 1 -- start start 2024-05-04T17:36:03.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f098128c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c0aafd0 con 0x7f097c0b48c0 2024-05-04T17:36:03.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f098128c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c0ab2b0 con 0x7f097c0a99c0 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f098128c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c0a9e80 con 0x7f097c13c350 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f097affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f097c0b48c0 0x7f097c13bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36480/0 (socket says 172.21.15.110:36480) 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f097affd640 1 -- 172.21.15.110:0/4148307935 learned_addr learned my addr 172.21.15.110:0/4148307935 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.215+0000 7f097b7fe640 1 --1- 172.21.15.110:0/4148307935 >> v1:172.21.15.110:6790/0 conn(0x7f097c13c350 0x7f097c146080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46748/0 (socket says 172.21.15.110:46748) 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1725962925 0 0) 0x7f097c0aafd0 con 0x7f097c0b48c0 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0964003520 con 0x7f097c0b48c0 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995697133 0 0) 0x7f097c0a9e80 con 0x7f097c13c350 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097c0aafd0 con 0x7f097c13c350 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975698586 0 0) 0x7f097c0ab2b0 con 0x7f097c0a99c0 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097c0a9e80 con 0x7f097c0a99c0 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 976163094 0 0) 0x7f0964003520 con 0x7f097c0b48c0 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 >> v1:172.21.15.110:6790/0 conn(0x7f097c13c350 legacy=0x7f097c146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 >> v1:172.21.15.136:6789/0 conn(0x7f097c0a99c0 legacy=0x7f097c145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c0b4ca0 con 0x7f097c0b48c0 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0974001610 con 0x7f097c0b48c0 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09740018b0 con 0x7f097c0b48c0 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f0979ffb640 1 -- 172.21.15.110:0/4148307935 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0974001cc0 con 0x7f097c0b48c0 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f098128c640 1 -- 172.21.15.110:0/4148307935 >> v1:172.21.15.110:6789/0 conn(0x7f097c0b48c0 legacy=0x7f097c13bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f098128c640 1 -- 172.21.15.110:0/4148307935 shutdown_connections 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.216+0000 7f098128c640 1 -- 172.21.15.110:0/4148307935 >> 172.21.15.110:0/4148307935 conn(0x7f097c072900 msgr2=0x7f097c072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.217+0000 7f098128c640 1 -- 172.21.15.110:0/4148307935 shutdown_connections 2024-05-04T17:36:03.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.217+0000 7f098128c640 1 -- 172.21.15.110:0/4148307935 wait complete. 2024-05-04T17:36:03.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f098128c640 1 Processor -- start 2024-05-04T17:36:03.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f098128c640 1 -- start start 2024-05-04T17:36:03.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f098128c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c1d7fe0 con 0x7f097c0a99c0 2024-05-04T17:36:03.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f098128c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c0abd20 con 0x7f097c13c350 2024-05-04T17:36:03.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f098128c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c1d73b0 con 0x7f097c0b48c0 2024-05-04T17:36:03.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f097a7fc640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f097c0b48c0 0x7f097c1d78d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46758/0 (socket says 172.21.15.110:46758) 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f097affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f097c0a99c0 0x7f097c1348e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36492/0 (socket says 172.21.15.110:36492) 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f097a7fc640 1 -- 172.21.15.110:0/3699307060 learned_addr learned my addr 172.21.15.110:0/3699307060 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2919853246 0 0) 0x7f097c1d7fe0 con 0x7f097c0a99c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0968003520 con 0x7f097c0a99c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.218+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897636028 0 0) 0x7f097c1d73b0 con 0x7f097c0b48c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097c1d7fe0 con 0x7f097c0b48c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910193094 0 0) 0x7f097c0abd20 con 0x7f097c13c350 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097c1d73b0 con 0x7f097c13c350 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2382182080 0 0) 0x7f0968003520 con 0x7f097c0a99c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f097c0abd20 con 0x7f097c0a99c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 491963786 0 0) 0x7f097c1d7fe0 con 0x7f097c0b48c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0968003520 con 0x7f097c0b48c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f09740047e0 con 0x7f097c0a99c0 2024-05-04T17:36:03.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0970001d50 con 0x7f097c0b48c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1283181321 0 0) 0x7f097c1d73b0 con 0x7f097c13c350 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f097c1d7fe0 con 0x7f097c13c350 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f096c0016b0 con 0x7f097c13c350 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 886437753 0 0) 0x7f097c0abd20 con 0x7f097c0a99c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 >> v1:172.21.15.110:6790/0 conn(0x7f097c0b48c0 legacy=0x7f097c1d78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 >> v1:172.21.15.136:6789/0 conn(0x7f097c13c350 legacy=0x7f097c12ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c1306e0 con 0x7f097c0a99c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f097c133b40 con 0x7f097c0a99c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f097c131230 con 0x7f097c0a99c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0974004ca0 con 0x7f097c0a99c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.219+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0974004ff0 con 0x7f097c0a99c0 2024-05-04T17:36:03.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.220+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097c0abb40 con 0x7f097c0a99c0 2024-05-04T17:36:03.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.220+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f097401b3c0 con 0x7f097c0a99c0 2024-05-04T17:36:03.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.221+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f0974086110 con 0x7f097c0a99c0 2024-05-04T17:36:03.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.225+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f097c0abb40 con 0x7f097c0a99c0 2024-05-04T17:36:03.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.401+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f097c0b6fe0 con 0x7f096806cad0 2024-05-04T17:36:03.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.402+0000 7f09637fe640 1 -- 172.21.15.110:0/3699307060 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35500 (unknown 2965378022 0 4196449677) 0x7f097c0b6fe0 con 0x7f096806cad0 2024-05-04T17:36:03.404 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:36:03.404 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:36:03.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.404+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 >> v1:172.21.15.110:6816/783516658 conn(0x7f096806cad0 legacy=0x7f096806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.404+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 >> v1:172.21.15.110:6789/0 conn(0x7f097c0a99c0 legacy=0x7f097c1348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.405+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 shutdown_connections 2024-05-04T17:36:03.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.405+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 >> 172.21.15.110:0/3699307060 conn(0x7f097c072900 msgr2=0x7f097c0b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:03.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.405+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 shutdown_connections 2024-05-04T17:36:03.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.405+0000 7f098128c640 1 -- 172.21.15.110:0/3699307060 wait complete. 2024-05-04T17:36:03.417 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-04T17:36:03.125178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":222944,"kb_used_data":2160,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751776,"statfs":{"total":773094113280,"available":772865818624,"internally_reserved":0,"allocated":2211840,"data_stored":1405824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.223029+0000","last_change":"2024-05-04T17:35:59.221521+0000","last_active":"2024-05-04T17:35:59.223029+0000","last_peered":"2024-05-04T17:35:59.223029+0000","last_clean":"2024-05-04T17:35:59.223029+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:35:59.223029+0000","last_undegraded":"2024-05-04T17:35:59.223029+0000","last_fullsized":"2024-05-04T17:35:59.223029+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:27:44.959711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:02.073258+0000","last_change":"2024-05-04T17:35:59.221461+0000","last_active":"2024-05-04T17:36:02.073258+0000","last_peered":"2024-05-04T17:36:02.073258+0000","last_clean":"2024-05-04T17:36:02.073258+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:36:02.073258+0000","last_undegraded":"2024-05-04T17:36:02.073258+0000","last_fullsized":"2024-05-04T17:36:02.073258+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:44:09.888594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.223837+0000","last_change":"2024-05-04T17:35:59.222186+0000","last_active":"2024-05-04T17:35:59.223837+0000","last_peered":"2024-05-04T17:35:59.223837+0000","last_clean":"2024-05-04T17:35:59.223837+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:35:59.223837+0000","last_undegraded":"2024-05-04T17:35:59.223837+0000","last_fullsized":"2024-05-04T17:35:59.223837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:40:36.227188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:02.073253+0000","last_change":"2024-05-04T17:35:59.218744+0000","last_active":"2024-05-04T17:36:02.073253+0000","last_peered":"2024-05-04T17:36:02.073253+0000","last_clean":"2024-05-04T17:36:02.073253+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:36:02.073253+0000","last_undegraded":"2024-05-04T17:36:02.073253+0000","last_fullsized":"2024-05-04T17:36:02.073253+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:37:18.872091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:00.220271+0000","last_change":"2024-05-04T17:35:59.223094+0000","last_active":"2024-05-04T17:36:00.220271+0000","last_peered":"2024-05-04T17:36:00.220271+0000","last_clean":"2024-05-04T17:36:00.220271+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:36:00.220271+0000","last_undegraded":"2024-05-04T17:36:00.220271+0000","last_fullsized":"2024-05-04T17:36:00.220271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:54:23.823020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:00.974105+0000","last_change":"2024-05-04T17:35:59.331384+0000","last_active":"2024-05-04T17:36:00.974105+0000","last_peered":"2024-05-04T17:36:00.974105+0000","last_clean":"2024-05-04T17:36:00.974105+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:36:00.974105+0000","last_undegraded":"2024-05-04T17:36:00.974105+0000","last_fullsized":"2024-05-04T17:36:00.974105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:24:22.601115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.224710+0000","last_change":"2024-05-04T17:35:59.223322+0000","last_active":"2024-05-04T17:35:59.224710+0000","last_peered":"2024-05-04T17:35:59.224710+0000","last_clean":"2024-05-04T17:35:59.224710+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:35:59.224710+0000","last_undegraded":"2024-05-04T17:35:59.224710+0000","last_fullsized":"2024-05-04T17:35:59.224710+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:23:10.183237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-04T17:36:00.220272+0000","last_change":"2024-05-04T17:35:59.221969+0000","last_active":"2024-05-04T17:36:00.220272+0000","last_peered":"2024-05-04T17:36:00.220272+0000","last_clean":"2024-05-04T17:36:00.220272+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:36:00.220272+0000","last_undegraded":"2024-05-04T17:36:00.220272+0000","last_fullsized":"2024-05-04T17:36:00.220272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:22:19.190308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-04T17:35:59.232135+0000","last_change":"2024-05-04T17:35:58.172775+0000","last_active":"2024-05-04T17:35:59.232135+0000","last_peered":"2024-05-04T17:35:59.232135+0000","last_clean":"2024-05-04T17:35:59.232135+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:35:59.232135+0000","last_undegraded":"2024-05-04T17:35:59.232135+0000","last_fullsized":"2024-05-04T17:35:59.232135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_clean_scrub_stamp":"2024-05-04T17:35:57.156954+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:43:38.793258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542146,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":671744,"data_stored":570110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":147456,"data_stored":45810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":41182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":574738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:36:03.417 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-04T17:36:03.417 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-04T17:36:03.418 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-04T17:36:03.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-04T17:36:03.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.511+0000 7f830408c640 1 Processor -- start 2024-05-04T17:36:03.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f830408c640 1 -- start start 2024-05-04T17:36:03.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f830408c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc0aa130 con 0x7f82fc0a96f0 2024-05-04T17:36:03.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f830408c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc0ab000 con 0x7f82fc0b4110 2024-05-04T17:36:03.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f830408c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc0a9c10 con 0x7f82fc0b4de0 2024-05-04T17:36:03.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f8301e28640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f82fc0b4de0 0x7f82fc13f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46766/0 (socket says 172.21.15.110:46766) 2024-05-04T17:36:03.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f8302629640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f82fc0a96f0 0x7f82fc14db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36504/0 (socket says 172.21.15.110:36504) 2024-05-04T17:36:03.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f8301e28640 1 -- 172.21.15.110:0/2435566943 learned_addr learned my addr 172.21.15.110:0/2435566943 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.512+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912715757 0 0) 0x7f82fc0a9c10 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e8003520 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228599573 0 0) 0x7f82fc0aa130 con 0x7f82fc0a96f0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82fc0a9c10 con 0x7f82fc0a96f0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 577742288 0 0) 0x7f82fc0ab000 con 0x7f82fc0b4110 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82fc0aa130 con 0x7f82fc0b4110 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 34145244 0 0) 0x7f82e8003520 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 >> v1:172.21.15.136:6789/0 conn(0x7f82fc0b4110 legacy=0x7f82fc142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 >> v1:172.21.15.110:6789/0 conn(0x7f82fc0a96f0 legacy=0x7f82fc14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc13fbb0 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f82f8001610 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f82f80018b0 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f8300e26640 1 -- 172.21.15.110:0/2435566943 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f82f8001cc0 con 0x7f82fc0b4de0 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f830408c640 1 -- 172.21.15.110:0/2435566943 >> v1:172.21.15.110:6790/0 conn(0x7f82fc0b4de0 legacy=0x7f82fc13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f830408c640 1 -- 172.21.15.110:0/2435566943 shutdown_connections 2024-05-04T17:36:03.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f830408c640 1 -- 172.21.15.110:0/2435566943 >> 172.21.15.110:0/2435566943 conn(0x7f82fc061e50 msgr2=0x7f82fc062260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:03.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.513+0000 7f830408c640 1 -- 172.21.15.110:0/2435566943 shutdown_connections 2024-05-04T17:36:03.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.514+0000 7f830408c640 1 -- 172.21.15.110:0/2435566943 wait complete. 2024-05-04T17:36:03.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.514+0000 7f830408c640 1 Processor -- start 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.514+0000 7f830408c640 1 -- start start 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f830408c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc138340 con 0x7f82fc0a96f0 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f830408c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc1d4f10 con 0x7f82fc0b4de0 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f830408c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc1d50e0 con 0x7f82fc1463a0 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f8301e28640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f82fc0a96f0 0x7f82fc1db2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36514/0 (socket says 172.21.15.110:36514) 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f8301e28640 1 -- 172.21.15.110:0/819111645 learned_addr learned my addr 172.21.15.110:0/819111645 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f8302629640 1 -- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6790/0 conn(0x7f82fc1463a0 legacy=0x7f82fc1d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f8302629640 1 -- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6790/0 conn(0x7f82fc1463a0 legacy=0x7f82fc1d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f8302629640 1 --1- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6790/0 conn(0x7f82fc1463a0 0x7f82fc1d4800 :-1 s=OPENED pgs=137 cs=1 l=1).handle_message read tag failed 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f8302629640 1 --1- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6790/0 conn(0x7f82fc1463a0 0x7f82fc1d4800 :-1 s=OPENED pgs=137 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322179998 0 0) 0x7f82fc138340 con 0x7f82fc0a96f0 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e0003520 con 0x7f82fc0a96f0 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213060049 0 0) 0x7f82fc1d4f10 con 0x7f82fc0b4de0 2024-05-04T17:36:03.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82fc138340 con 0x7f82fc0b4de0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2383791989 0 0) 0x7f82e0003520 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82fc1d4f10 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.515+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f82f8001d00 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 825716848 0 0) 0x7f82fc1d4f10 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6790/0 conn(0x7f82fc1463a0 legacy=0x7f82fc1d4800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 >> v1:172.21.15.136:6789/0 conn(0x7f82fc0b4de0 legacy=0x7f82fc1d40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc1dacf0 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f82fc1dafe0 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f82fc139000 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f82f80018b0 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.516+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f82f8004d50 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.517+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e4005080 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.517+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f82f8004ef0 con 0x7f82fc0a96f0 2024-05-04T17:36:03.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.518+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f82f80861c0 con 0x7f82fc0a96f0 2024-05-04T17:36:03.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.522+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f82f8055190 con 0x7f82fc0a96f0 2024-05-04T17:36:03.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.734+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f82e4005370 con 0x7f82fc0a96f0 2024-05-04T17:36:03.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.734+0000 7f82f27fc640 1 -- 172.21.15.110:0/819111645 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f82f8054a90 con 0x7f82fc0a96f0 2024-05-04T17:36:03.736 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:36:03.736 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-04T17:36:03.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.736+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6816/783516658 conn(0x7f82e006c4f0 legacy=0x7f82e006e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.736+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 >> v1:172.21.15.110:6789/0 conn(0x7f82fc0a96f0 legacy=0x7f82fc1db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.736+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 shutdown_connections 2024-05-04T17:36:03.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.736+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 >> 172.21.15.110:0/819111645 conn(0x7f82fc061e50 msgr2=0x7f82fc0b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:03.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.736+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 shutdown_connections 2024-05-04T17:36:03.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.736+0000 7f830408c640 1 -- 172.21.15.110:0/819111645 wait complete. 2024-05-04T17:36:03.748 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-04T17:36:03.749 INFO:teuthology.run_tasks:Running task exec... 2024-05-04T17:36:03.758 INFO:teuthology.task.exec:Executing custom commands... 2024-05-04T17:36:03.759 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:36:03.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd pool create cephsqlite' 2024-05-04T17:36:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.852+0000 7f427c52b640 1 Processor -- start 2024-05-04T17:36:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.852+0000 7f427c52b640 1 -- start start 2024-05-04T17:36:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f427c52b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4274071ab0 con 0x7f427413c960 2024-05-04T17:36:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f427c52b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740730c0 con 0x7f42740aea60 2024-05-04T17:36:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f427c52b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4274071d20 con 0x7f427406f980 2024-05-04T17:36:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f4279ac6640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f427406f980 0x7f427413c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46782/0 (socket says 172.21.15.110:46782) 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f427aac8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f427413c960 0x7f427414a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36526/0 (socket says 172.21.15.110:36526) 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f4279ac6640 1 -- 172.21.15.110:0/759682841 learned_addr learned my addr 172.21.15.110:0/759682841 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101172227 0 0) 0x7f4274071ab0 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4260003520 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720629566 0 0) 0x7f4274071d20 con 0x7f427406f980 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4274071ab0 con 0x7f427406f980 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3896385147 0 0) 0x7f4260003520 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 >> v1:172.21.15.110:6790/0 conn(0x7f427406f980 legacy=0x7f427413c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 >> v1:172.21.15.136:6789/0 conn(0x7f42740aea60 legacy=0x7f42740729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4274072420 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f426c001990 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f426c001c70 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.853+0000 7f42792c5640 1 -- 172.21.15.110:0/759682841 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f426c002040 con 0x7f427413c960 2024-05-04T17:36:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.854+0000 7f427c52b640 1 -- 172.21.15.110:0/759682841 >> v1:172.21.15.110:6789/0 conn(0x7f427413c960 legacy=0x7f427414a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.854+0000 7f427c52b640 1 -- 172.21.15.110:0/759682841 shutdown_connections 2024-05-04T17:36:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.854+0000 7f427c52b640 1 -- 172.21.15.110:0/759682841 >> 172.21.15.110:0/759682841 conn(0x7f427406bc70 msgr2=0x7f42740b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.854+0000 7f427c52b640 1 -- 172.21.15.110:0/759682841 shutdown_connections 2024-05-04T17:36:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.854+0000 7f427c52b640 1 -- 172.21.15.110:0/759682841 wait complete. 2024-05-04T17:36:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.855+0000 7f427c52b640 1 Processor -- start 2024-05-04T17:36:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.855+0000 7f427c52b640 1 -- start start 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f427c52b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740aa100 con 0x7f427406f980 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f427c52b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740adf60 con 0x7f427413c960 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f427c52b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4274140040 con 0x7f42740aea60 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f4279ac6640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f42740aea60 0x7f427413f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46790/0 (socket says 172.21.15.110:46790) 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f4279ac6640 1 -- 172.21.15.110:0/2183935062 learned_addr learned my addr 172.21.15.110:0/2183935062 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f427a2c7640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f427406f980 0x7f427413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36542/0 (socket says 172.21.15.110:36542) 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3803671619 0 0) 0x7f4274140040 con 0x7f42740aea60 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4258005730 con 0x7f42740aea60 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190863435 0 0) 0x7f42740aa100 con 0x7f427406f980 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4274140040 con 0x7f427406f980 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 750742726 0 0) 0x7f42740adf60 con 0x7f427413c960 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42740aa100 con 0x7f427413c960 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2679509521 0 0) 0x7f4274140040 con 0x7f427406f980 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42740adf60 con 0x7f427406f980 2024-05-04T17:36:03.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f42700018b0 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3817201439 0 0) 0x7f4258005730 con 0x7f42740aea60 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4274140040 con 0x7f42740aea60 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4264001850 con 0x7f42740aea60 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4040359367 0 0) 0x7f42740adf60 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 >> v1:172.21.15.110:6790/0 conn(0x7f42740aea60 legacy=0x7f427413f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 >> v1:172.21.15.136:6789/0 conn(0x7f427413c960 legacy=0x7f42741def70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42741d7960 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42741d7b50 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f427413f070 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4270001bd0 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.856+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4270003a60 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.857+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f425c005080 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.858+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f4270003c80 con 0x7f427406f980 2024-05-04T17:36:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.858+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 3143449792 0 0) 0x7f42700860f0 con 0x7f427406f980 2024-05-04T17:36:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:03.863+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f42700550c0 con 0x7f427406f980 2024-05-04T17:36:04.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.036+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cephsqlite"} v 0) -- 0x7f425c005370 con 0x7f427406f980 2024-05-04T17:36:04.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.240+0000 7f426affd640 1 -- 172.21.15.110:0/2183935062 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephsqlite"}]=0 pool 'cephsqlite' created v18) ==== 110+0+0 (unknown 1838427875 0 0) 0x7f42700549c0 con 0x7f427406f980 2024-05-04T17:36:04.242 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephsqlite' created 2024-05-04T17:36:04.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.243+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 >> v1:172.21.15.110:6816/783516658 conn(0x7f425806e9b0 legacy=0x7f4258070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.243+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 >> v1:172.21.15.110:6789/0 conn(0x7f427406f980 legacy=0x7f427413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.244+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 shutdown_connections 2024-05-04T17:36:04.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.244+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 >> 172.21.15.110:0/2183935062 conn(0x7f427406bc70 msgr2=0x7f4274074b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:04.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.244+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 shutdown_connections 2024-05-04T17:36:04.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.244+0000 7f427c52b640 1 -- 172.21.15.110:0/2183935062 wait complete. 2024-05-04T17:36:04.259 DEBUG:teuthology.orchestra.run.smithi110:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph auth get-or-create client.libcephsqlite mon '"'"'profile simple-rados-client-with-blocklist'"'"' osd '"'"'allow rwx pool=cephsqlite'"'"' >> /etc/ceph/ceph.keyring' 2024-05-04T17:36:04.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.356+0000 7ff3a344e640 1 Processor -- start 2024-05-04T17:36:04.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a344e640 1 -- start start 2024-05-04T17:36:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a344e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c142290 con 0x7ff39c0b5240 2024-05-04T17:36:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a344e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c0ab370 con 0x7ff39c0b4540 2024-05-04T17:36:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a344e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c13c660 con 0x7ff39c0a9b40 2024-05-04T17:36:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a11ea640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff39c0b5240 0x7ff39c13bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36562/0 (socket says 172.21.15.110:36562) 2024-05-04T17:36:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a09e9640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7ff39c0a9b40 0x7ff39c145ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:46802/0 (socket says 172.21.15.110:46802) 2024-05-04T17:36:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a11ea640 1 -- 172.21.15.110:0/1425676510 learned_addr learned my addr 172.21.15.110:0/1425676510 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.357+0000 7ff3a11ea640 1 -- 172.21.15.110:0/1425676510 >> v1:172.21.15.110:6789/0 conn(0x7ff39c0b5240 legacy=0x7ff39c13bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606397606 0 0) 0x7ff39c13c660 con 0x7ff39c0a9b40 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff388003520 con 0x7ff39c0a9b40 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023805104 0 0) 0x7ff39c0ab370 con 0x7ff39c0b4540 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff39c13c660 con 0x7ff39c0b4540 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 740783497 0 0) 0x7ff388003520 con 0x7ff39c0a9b40 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 >> v1:172.21.15.136:6789/0 conn(0x7ff39c0b4540 legacy=0x7ff39c0b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 >> v1:172.21.15.110:6789/0 conn(0x7ff39c0b5240 legacy=0x7ff39c13bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c1463b0 con 0x7ff39c0a9b40 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff38c001610 con 0x7ff39c0a9b40 2024-05-04T17:36:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff38c001930 con 0x7ff39c0a9b40 2024-05-04T17:36:04.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff393fff640 1 -- 172.21.15.110:0/1425676510 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff38c001d00 con 0x7ff39c0a9b40 2024-05-04T17:36:04.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.358+0000 7ff3a344e640 1 -- 172.21.15.110:0/1425676510 >> v1:172.21.15.110:6790/0 conn(0x7ff39c0a9b40 legacy=0x7ff39c145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.359+0000 7ff3a344e640 1 -- 172.21.15.110:0/1425676510 shutdown_connections 2024-05-04T17:36:04.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.359+0000 7ff3a344e640 1 -- 172.21.15.110:0/1425676510 >> 172.21.15.110:0/1425676510 conn(0x7ff39c06d580 msgr2=0x7ff39c06d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:04.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.359+0000 7ff3a344e640 1 -- 172.21.15.110:0/1425676510 shutdown_connections 2024-05-04T17:36:04.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.359+0000 7ff3a344e640 1 -- 172.21.15.110:0/1425676510 wait complete. 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a344e640 1 Processor -- start 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a344e640 1 -- start start 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a344e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c0b0020 con 0x7ff39c0a9b40 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a344e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c0b06b0 con 0x7ff39c0b4540 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a344e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39c0b0290 con 0x7ff39c0b5240 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a11ea640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff39c0a9b40 0x7ff39c0adea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36570/0 (socket says 172.21.15.110:36570) 2024-05-04T17:36:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff3a11ea640 1 -- 172.21.15.110:0/1693316274 learned_addr learned my addr 172.21.15.110:0/1693316274 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369714533 0 0) 0x7ff39c0b0290 con 0x7ff39c0b5240 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff380003520 con 0x7ff39c0b5240 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2061309812 0 0) 0x7ff39c0b0020 con 0x7ff39c0a9b40 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.360+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff39c0b0290 con 0x7ff39c0a9b40 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278218986 0 0) 0x7ff39c1463b0 con 0x7ff39c0b4540 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff39c0b0020 con 0x7ff39c0b4540 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1863640152 0 0) 0x7ff380003520 con 0x7ff39c0b5240 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff39c1463b0 con 0x7ff39c0b5240 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff394001bd0 con 0x7ff39c0b5240 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 973795080 0 0) 0x7ff39c0b0290 con 0x7ff39c0a9b40 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff380003520 con 0x7ff39c0a9b40 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff3980017e0 con 0x7ff39c0a9b40 2024-05-04T17:36:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1544899457 0 0) 0x7ff39c0b0020 con 0x7ff39c0b4540 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff39c0b0290 con 0x7ff39c0b4540 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff38c0015d0 con 0x7ff39c0b4540 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1642577508 0 0) 0x7ff39c1463b0 con 0x7ff39c0b5240 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 >> v1:172.21.15.136:6789/0 conn(0x7ff39c0b4540 legacy=0x7ff39c13b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 >> v1:172.21.15.110:6789/0 conn(0x7ff39c0a9b40 legacy=0x7ff39c0adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39c13b800 con 0x7ff39c0b5240 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff39c13baf0 con 0x7ff39c0b5240 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff394003990 con 0x7ff39c0b5240 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7ff394003ce0 con 0x7ff39c0b5240 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.361+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff39c0aed20 con 0x7ff39c0b5240 2024-05-04T17:36:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.362+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39c04fe40 con 0x7ff39c0b5240 2024-05-04T17:36:04.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.362+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7ff394019e50 con 0x7ff39c0b5240 2024-05-04T17:36:04.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.363+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4439+0+0 (unknown 794485150 0 0) 0x7ff394085170 con 0x7ff39c0b5240 2024-05-04T17:36:04.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.367+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7ff394053fe0 con 0x7ff39c0b5240 2024-05-04T17:36:04.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.607+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.libcephsqlite", "caps": ["mon", "profile simple-rados-client-with-blocklist", "osd", "allow rwx pool=cephsqlite"]} v 0) -- 0x7ff39c06a700 con 0x7ff39c0b5240 2024-05-04T17:36:04.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.610+0000 7ff391ffb640 1 -- 172.21.15.110:0/1693316274 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.libcephsqlite", "caps": ["mon", "profile simple-rados-client-with-blocklist", "osd", "allow rwx pool=cephsqlite"]}]=0 v15) ==== 199+0+71 (unknown 3040008346 0 1820004929) 0x7ff3940538e0 con 0x7ff39c0b5240 2024-05-04T17:36:04.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.613+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 >> v1:172.21.15.110:6816/783516658 conn(0x7ff38006cab0 legacy=0x7ff38006ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.613+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 >> v1:172.21.15.110:6790/0 conn(0x7ff39c0b5240 legacy=0x7ff39c1d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:36:04.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.613+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 shutdown_connections 2024-05-04T17:36:04.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.613+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 >> 172.21.15.110:0/1693316274 conn(0x7ff39c06d580 msgr2=0x7ff39c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:36:04.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.613+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 shutdown_connections 2024-05-04T17:36:04.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:36:04.613+0000 7ff3a344e640 1 -- 172.21.15.110:0/1693316274 wait complete. 2024-05-04T17:36:04.625 INFO:teuthology.run_tasks:Running task exec... 2024-05-04T17:36:04.637 INFO:teuthology.task.exec:Executing custom commands... 2024-05-04T17:36:04.638 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:36:04.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr' 2024-05-04T17:36:04.680 INFO:teuthology.orchestra.run.smithi110.stdout:[==========] Running 20 tests from 1 test suite. 2024-05-04T17:36:04.680 INFO:teuthology.orchestra.run.smithi110.stdout:[----------] Global test environment set-up. 2024-05-04T17:36:04.680 INFO:teuthology.orchestra.run.smithi110.stdout:[----------] 20 tests from CephSQLiteTest 2024-05-04T17:36:09.687 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.Create 2024-05-04T17:36:09.698 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/0e20b0e0-8a15-4188-a1b7-e57fa029ac3e.db?vfs=ceph 2024-05-04T17:36:09.698 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:09.703 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a INT); 2024-05-04T17:36:09.726 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.Create (39 ms) 2024-05-04T17:36:09.726 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertBulk4096 2024-05-04T17:36:09.735 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/f04d182c-c00f-489b-a012-3596d5920f40.db?vfs=ceph 2024-05-04T17:36:09.735 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:09.741 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA page_size = 4096;CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOM() FROM c LIMIT 1000000;PRAGMA page_size; 2024-05-04T17:36:11.924 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertBulk4096 (2198 ms) 2024-05-04T17:36:11.924 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertBulk 2024-05-04T17:36:11.932 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/9ff0d9e8-5c12-4a69-bddb-1fe439432e97.db?vfs=ceph 2024-05-04T17:36:11.932 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:11.938 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOM() FROM c LIMIT 1000000; 2024-05-04T17:36:12.387 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertBulk (463 ms) 2024-05-04T17:36:12.387 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.UpdateBulk 2024-05-04T17:36:12.394 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/bd46b64a-d71f-4c3e-8da4-0d35955e1e42.db?vfs=ceph 2024-05-04T17:36:12.394 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:12.398 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT x FROM c LIMIT 1000000;SELECT SUM(a) FROM foo;UPDATE foo SET a = a+a;SELECT SUM(a) FROM foo; 2024-05-04T17:36:13.104 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.UpdateBulk (717 ms) 2024-05-04T17:36:13.104 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertRate 2024-05-04T17:36:13.111 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/6e5d2503-c220-4ae1-be49-3d2490598f16.db?vfs=ceph 2024-05-04T17:36:13.111 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:13.115 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM()); 2024-05-04T17:36:14.779 INFO:teuthology.orchestra.run.smithi110.stdout:transactions per second: 60.4971 2024-05-04T17:36:14.779 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertRate (1676 ms) 2024-05-04T17:36:14.780 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.DatabaseShrink 2024-05-04T17:36:14.787 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/43cba41b-85c9-40f1-9e35-eb925c78321d.db?vfs=ceph 2024-05-04T17:36:14.787 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:14.791 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT x FROM c LIMIT 1000000;DELETE FROM foo WHERE RANDOM()%4 < 3;VACUUM; 2024-05-04T17:36:16.057 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.DatabaseShrink (1277 ms) 2024-05-04T17:36:16.058 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertExclusiveRate 2024-05-04T17:36:16.065 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/c30bf16e-c1b5-42d7-9954-0f51c0a4224b.db?vfs=ceph 2024-05-04T17:36:16.065 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:16.068 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA locking_mode=EXCLUSIVE;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM()); 2024-05-04T17:36:16.997 INFO:teuthology.orchestra.run.smithi110.stdout:transactions per second: 109.411 2024-05-04T17:36:17.000 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertExclusiveRate (943 ms) 2024-05-04T17:36:17.000 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertExclusiveWALRate 2024-05-04T17:36:17.006 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/01a33d9b-3712-4365-ae03-7514c22476f5.db?vfs=ceph 2024-05-04T17:36:17.007 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:17.012 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA locking_mode=EXCLUSIVE;PRAGMA journal_mode=WAL;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM()); 2024-05-04T17:36:17.470 INFO:teuthology.orchestra.run.smithi110.stdout:transactions per second: 240.968 2024-05-04T17:36:17.486 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertExclusiveWALRate (486 ms) 2024-05-04T17:36:17.486 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.WALTransactionSync 2024-05-04T17:36:17.494 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/c65c834b-0d08-464b-b638-3801562a51b4.db?vfs=ceph 2024-05-04T17:36:17.494 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:17.497 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA locking_mode=EXCLUSIVE;PRAGMA journal_mode=WAL;CREATE TABLE foo (a INT);INSERT INTO perf (v) VALUES (ceph_perf());BEGIN TRANSACTION;INSERT INTO foo (a) VALUES (RANDOM());END TRANSACTION;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount' AND b.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount'; 2024-05-04T17:36:17.567 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.WALTransactionSync (81 ms) 2024-05-04T17:36:17.567 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.PersistTransactionSync 2024-05-04T17:36:17.574 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/66542a8e-c4c3-46b6-90d7-0e9ed283ecce.db?vfs=ceph 2024-05-04T17:36:17.574 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:17.579 INFO:teuthology.orchestra.run.smithi110.stdout:BEGIN TRANSACTION;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM());END TRANSACTION;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount' AND b.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount'; 2024-05-04T17:36:17.597 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.PersistTransactionSync (30 ms) 2024-05-04T17:36:17.597 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertExclusiveLock 2024-05-04T17:36:17.604 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/1686ff20-002d-4777-912b-ba8a75b0b5fd.db?vfs=ceph 2024-05-04T17:36:17.604 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:17.607 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA locking_mode=EXCLUSIVE;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM());INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_vfs.opf_lock.avgcount' AND b.fullkey = '$.libcephsqlite_vfs.opf_lock.avgcount';SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.lock' AND b.fullkey = '$.libcephsqlite_striper.lock'; 2024-05-04T17:36:17.625 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertExclusiveLock (28 ms) 2024-05-04T17:36:17.625 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.TransactionSizeUpdate 2024-05-04T17:36:17.632 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/717d2342-ee7d-475e-a08c-4102f7d8ed1f.db?vfs=ceph 2024-05-04T17:36:17.632 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:17.635 INFO:teuthology.orchestra.run.smithi110.stdout:BEGIN TRANSACTION;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM());END TRANSACTION;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.update_size' AND b.fullkey = '$.libcephsqlite_striper.update_size'; 2024-05-04T17:36:17.658 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.TransactionSizeUpdate (33 ms) 2024-05-04T17:36:17.658 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.AllocatedGrowth 2024-05-04T17:36:17.665 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/7ef77990-cbcd-4779-a333-b57c3155fb17.db?vfs=ceph 2024-05-04T17:36:17.665 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:17.668 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a BLOB);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOMBLOB(1<<20) FROM c LIMIT 1024;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.update_allocated' AND b.fullkey = '$.libcephsqlite_striper.update_allocated'; 2024-05-04T17:36:25.002 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.AllocatedGrowth (7344 ms) 2024-05-04T17:36:25.002 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.DeleteBulk 2024-05-04T17:36:25.010 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/3bc70d38-8de0-43d4-ad29-66f8aaecac57.db?vfs=ceph 2024-05-04T17:36:25.010 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:25.014 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT x FROM c LIMIT 1000000;DELETE FROM foo WHERE RANDOM()%2 == 0; 2024-05-04T17:36:25.861 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.DeleteBulk (859 ms) 2024-05-04T17:36:25.862 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.DropMassive 2024-05-04T17:36:25.869 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/b2c4c84b-c1a9-4d07-b9c9-fc1dba27cefc.db?vfs=ceph 2024-05-04T17:36:25.869 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:25.872 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a BLOB);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOMBLOB(1<<20) FROM c LIMIT 1024;DROP TABLE foo;VACUUM;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.shrink' AND b.fullkey = '$.libcephsqlite_striper.shrink';SELECT a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.shrink_bytes' AND b.fullkey = '$.libcephsqlite_striper.shrink_bytes'; 2024-05-04T17:36:55.661 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.DropMassive (29800 ms) 2024-05-04T17:36:55.661 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.InsertMassiveVerify 2024-05-04T17:36:55.669 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/1c3aea0c-f6cf-42c1-912b-4568d282c335.db?vfs=ceph 2024-05-04T17:36:55.669 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:36:55.673 INFO:teuthology.orchestra.run.smithi110.stdout:CREATE TABLE foo (a BLOB);CREATE TEMPORARY TABLE bar (a BLOB);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO bar (a) SELECT RANDOMBLOB(1<<20) FROM c LIMIT 1024;SELECT a FROM bar;INSERT INTO foo (a) SELECT a FROM bar;SELECT a FROM foo; 2024-05-04T17:37:09.029 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.InsertMassiveVerify (13368 ms) 2024-05-04T17:37:09.030 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.PerfValid 2024-05-04T17:37:09.038 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/e497407a-00df-4ffc-ac05-2bd5111720b8.db?vfs=ceph 2024-05-04T17:37:09.039 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:37:09.044 INFO:teuthology.orchestra.run.smithi110.stdout:SELECT json_valid(ceph_perf()); 2024-05-04T17:37:09.045 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.PerfValid (16 ms) 2024-05-04T17:37:09.045 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.StatusValid 2024-05-04T17:37:09.054 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/ae1c7777-517a-4ed7-a3b8-a475290fe011.db?vfs=ceph 2024-05-04T17:37:09.054 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:37:09.059 INFO:teuthology.orchestra.run.smithi110.stdout:SELECT json_valid(ceph_status()); 2024-05-04T17:37:09.060 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.StatusValid (15 ms) 2024-05-04T17:37:09.060 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.CurrentTime 2024-05-04T17:37:09.068 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/beb46329-3c7e-430c-aace-f38a3ef8a594.db?vfs=ceph 2024-05-04T17:37:09.069 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:37:09.073 INFO:teuthology.orchestra.run.smithi110.stdout:SELECT strftime('%s', 'now'); 2024-05-04T17:37:09.074 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.CurrentTime (14 ms) 2024-05-04T17:37:09.074 INFO:teuthology.orchestra.run.smithi110.stdout:[ RUN ] CephSQLiteTest.StatusFields 2024-05-04T17:37:09.083 INFO:teuthology.orchestra.run.smithi110.stdout:using database: file:cephsqlite:/bb7ea2aa-7b78-4d22-8bfb-da4f6e4d3085.db?vfs=ceph 2024-05-04T17:37:09.083 INFO:teuthology.orchestra.run.smithi110.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-05-04T17:37:09.090 INFO:teuthology.orchestra.run.smithi110.stdout:SELECT json_extract(ceph_status(), '$.addr');SELECT json_extract(ceph_status(), '$.id'); 2024-05-04T17:37:09.090 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110:0/2962580428 2024-05-04T17:37:09.090 INFO:teuthology.orchestra.run.smithi110.stdout:4425 2024-05-04T17:37:09.091 INFO:teuthology.orchestra.run.smithi110.stdout:[ OK ] CephSQLiteTest.StatusFields (17 ms) 2024-05-04T17:37:09.091 INFO:teuthology.orchestra.run.smithi110.stdout:[----------] 20 tests from CephSQLiteTest (59404 ms total) 2024-05-04T17:37:09.091 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:37:09.091 INFO:teuthology.orchestra.run.smithi110.stdout:[----------] Global test environment tear-down 2024-05-04T17:37:09.091 INFO:teuthology.orchestra.run.smithi110.stdout:[==========] 20 tests from 1 test suite ran. (64411 ms total) 2024-05-04T17:37:09.091 INFO:teuthology.orchestra.run.smithi110.stdout:[ PASSED ] 20 tests. 2024-05-04T17:37:09.134 INFO:teuthology.run_tasks:Running task workunit... 2024-05-04T17:37:09.149 INFO:tasks.workunit:Pulling workunits from ref f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:37:09.149 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-04T17:37:09.150 DEBUG:teuthology.orchestra.run.smithi110:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-04T17:37:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:37:09.189 INFO:teuthology.orchestra.run.smithi110.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-04T17:37:09.190 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-04T17:37:09.244 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-04T17:37:09.244 DEBUG:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-04T17:37:09.298 INFO:tasks.workunit:timeout=3h 2024-05-04T17:37:09.298 INFO:tasks.workunit:cleanup=True 2024-05-04T17:37:09.299 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:37:09.353 INFO:tasks.workunit.client.0.smithi110.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-04T17:37:55.639 INFO:tasks.workunit.client.0.smithi110.stderr:Note: switching to 'f273489c6dcd4bc88409993babd09dd99491162c'. 2024-05-04T17:37:55.639 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.639 INFO:tasks.workunit.client.0.smithi110.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-04T17:37:55.639 INFO:tasks.workunit.client.0.smithi110.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr:state without impacting any branches by switching back to a branch. 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: git switch -c 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr:Or undo this operation with: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: git switch - 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:55.640 INFO:tasks.workunit.client.0.smithi110.stderr:HEAD is now at f273489c6dc Merge branch 'wip-65185-squid' of https://github.com/rzarzynski/ceph into wip-yuri11-testing-2024-05-02-1242-squid 2024-05-04T17:37:55.644 DEBUG:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-04T17:37:55.770 INFO:tasks.workunit.client.0.smithi110.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-04T17:37:55.772 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-04T17:37:55.773 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-04T17:37:56.338 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-04T17:37:56.395 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-04T17:37:56.435 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-04T17:37:56.437 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-04T17:37:56.437 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-04T17:37:56.476 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-04T17:37:56.479 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-04T17:37:56.479 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-04T17:37:56.494 INFO:tasks.workunit:Running workunits matching rados/test_libcephsqlite.sh on client.0... 2024-05-04T17:37:56.495 INFO:tasks.workunit:Running workunit rados/test_libcephsqlite.sh... 2024-05-04T17:37:56.495 DEBUG:teuthology.orchestra.run.smithi110:workunit test rados/test_libcephsqlite.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f273489c6dcd4bc88409993babd09dd99491162c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CEPH_ARGS='--id libcephsqlite --no-log-to-stderr' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh cephsqlite 2024-05-04T17:37:56.552 INFO:tasks.workunit.client.0.smithi110.stderr:+ pool=cephsqlite 2024-05-04T17:37:56.553 INFO:tasks.workunit.client.0.smithi110.stderr:++ basename /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh 2024-05-04T17:37:56.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ ns=test_libcephsqlite.sh 2024-05-04T17:37:56.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ striper rm baz.db 2024-05-04T17:37:56.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-05-04T17:37:56.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:+ background= 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' '' = b ']' 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:++ cat 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:+ a='create table if not exists foo (a INT); 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:insert into foo (a) values (RANDOM()); 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:drop table foo;' 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:+ printf %s 'create table if not exists foo (a INT); 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:insert into foo (a) values (RANDOM()); 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:drop table foo;' 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:create table if not exists foo (a INT); 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:insert into foo (a) values (RANDOM()); 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:drop table foo;+ '[' '' '!=' b ']' 2024-05-04T17:37:56.596 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 2024-05-04T17:37:56.597 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:37:56.795 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:56.795 INFO:tasks.workunit.client.0.smithi110.stderr:real 0m0.201s 2024-05-04T17:37:56.795 INFO:tasks.workunit.client.0.smithi110.stderr:user 0m0.035s 2024-05-04T17:37:56.795 INFO:tasks.workunit.client.0.smithi110.stderr:sys 0m0.019s 2024-05-04T17:37:56.795 INFO:tasks.workunit.client.0.smithi110.stderr:+ striper stat baz.db 2024-05-04T17:37:56.795 INFO:tasks.workunit.client.0.smithi110.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper stat baz.db 2024-05-04T17:37:56.827 INFO:tasks.workunit.client.0.smithi110.stdout:cephsqlite/baz.db mtime 2024-05-04T17:37:56.000000+0000, size 8192 2024-05-04T17:37:56.829 INFO:tasks.workunit.client.0.smithi110.stderr:+ striper rm baz.db 2024-05-04T17:37:56.829 INFO:tasks.workunit.client.0.smithi110.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ repeat 10 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ n=10 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ shift 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i = 0 )) 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 10 )) 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:+ background= 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' '' = b ']' 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ cat 2024-05-04T17:37:56.872 INFO:tasks.workunit.client.0.smithi110.stderr:+ a='CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT LENGTH(text) FROM rand; 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:DROP TABLE rand;' 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:+ printf %s 'CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.873 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT LENGTH(text) FROM rand; 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:DROP TABLE rand;' 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-05-04T17:37:56.874 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT LENGTH(text) FROM rand; 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:DROP TABLE rand;+ '[' '' '!=' b ']' 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 2024-05-04T17:37:56.875 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:37:57.483 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.483 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.483 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.483 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.483 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.483 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.484 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.484 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.484 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.484 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.604 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:real 0m0.726s 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:user 0m0.077s 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:sys 0m0.039s 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ repeat 100 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ n=100 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ shift 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i = 0 )) 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.605 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.606 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.611 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.612 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.613 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.615 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.616 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.617 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.618 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.619 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.620 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.621 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( ++i )) 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:++ (( i < 100 )) 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:+ background= 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' '' = b ']' 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:++ cat 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:+ a='BEGIN TRANSACTION; 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.627 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.628 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.629 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.630 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.631 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.632 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.633 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:COMMIT; 2024-05-04T17:37:57.634 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT LENGTH(text) FROM rand; 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:DROP TABLE rand;' 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:+ printf %s 'BEGIN TRANSACTION; 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.635 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.636 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.637 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.638 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.639 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.640 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.641 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:COMMIT; 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT LENGTH(text) FROM rand; 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:DROP TABLE rand;' 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:BEGIN TRANSACTION; 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-05-04T17:37:57.642 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.643 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.644 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.645 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.646 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.647 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.648 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:COMMIT; 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT LENGTH(text) FROM rand; 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:DROP TABLE rand;+ '[' '' '!=' b ']' 2024-05-04T17:37:57.649 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 2024-05-04T17:37:57.650 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:37:57.822 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.822 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.822 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.823 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.824 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.825 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.826 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.826 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.826 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.826 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.826 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.827 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.828 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.829 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:57.830 INFO:tasks.workunit.client.0.smithi110.stdout:4096 2024-05-04T17:37:58.513 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-05-04T17:37:58.513 INFO:tasks.workunit.client.0.smithi110.stderr:real 0m0.920s 2024-05-04T17:37:58.513 INFO:tasks.workunit.client.0.smithi110.stderr:user 0m0.089s 2024-05-04T17:37:58.513 INFO:tasks.workunit.client.0.smithi110.stderr:sys 0m0.046s 2024-05-04T17:37:58.513 INFO:tasks.workunit.client.0.smithi110.stderr:+ striper rm baz.db 2024-05-04T17:37:58.513 INFO:tasks.workunit.client.0.smithi110.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-05-04T17:37:58.551 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:37:58.552 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:37:58 PM UTC 2024 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite b 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ background=b 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' b = b ']' 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ shift 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:++ cat 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ a='CREATE TABLE foo (a BLOB); 2024-05-04T17:37:58.553 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo VALUES ("start"); 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr:WITH RECURSIVE c(x) AS 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: ( 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: VALUES(1) 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: UNION ALL 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT x+1 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: ) 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo (a) 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT RANDOMBLOB(1<<20) 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr: LIMIT (1<<20);' 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ printf %s 'CREATE TABLE foo (a BLOB); 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo VALUES ("start"); 2024-05-04T17:37:58.554 INFO:tasks.workunit.client.0.smithi110.stderr:WITH RECURSIVE c(x) AS 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: ( 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: VALUES(1) 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: UNION ALL 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT x+1 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: ) 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo (a) 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT RANDOMBLOB(1<<20) 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: LIMIT (1<<20);' 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr:CREATE TABLE foo (a BLOB); 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo VALUES ("start"); 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr:WITH RECURSIVE c(x) AS 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: ( 2024-05-04T17:37:58.555 INFO:tasks.workunit.client.0.smithi110.stderr: VALUES(1) 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: UNION ALL 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT x+1 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: ) 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo (a) 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT RANDOMBLOB(1<<20) 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr: LIMIT (1<<20);+ '[' b '!=' b ']' 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr:+ sleep 10 2024-05-04T17:37:58.556 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:38:08.554 INFO:tasks.workunit.client.0.smithi110.stdout:done 2024-05-04T17:38:08.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo done 2024-05-04T17:38:08.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ jobs -l 2024-05-04T17:38:08.555 INFO:tasks.workunit.client.0.smithi110.stdout:[1]+ 36505 Running sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a" & 2024-05-04T17:38:08.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ jobs -p 2024-05-04T17:38:08.555 INFO:tasks.workunit.client.0.smithi110.stderr:+ kill -KILL -- 36505 2024-05-04T17:38:08.555 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:38:08.556 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:38:08 PM UTC 2024 2024-05-04T17:38:08.556 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 2024-05-04T17:38:08.556 INFO:tasks.workunit.client.0.smithi110.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh: line 95: 36505 Killed sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a" 2024-05-04T17:38:08.556 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:38:08.557 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:38:08 PM UTC 2024 2024-05-04T17:38:08.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ sqlite 2024-05-04T17:38:08.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ background= 2024-05-04T17:38:08.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' = b ']' 2024-05-04T17:38:08.558 INFO:tasks.workunit.client.0.smithi110.stderr:+++ cat 2024-05-04T17:38:08.559 INFO:tasks.workunit.client.0.smithi110.stderr:++ a='SELECT COUNT(*) FROM foo;' 2024-05-04T17:38:08.559 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %s 'SELECT COUNT(*) FROM foo;' 2024-05-04T17:38:08.559 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT COUNT(*) FROM foo;++ '[' '' '!=' b ']' 2024-05-04T17:38:08.559 INFO:tasks.workunit.client.0.smithi110.stderr:++ wait 2024-05-04T17:38:08.559 INFO:tasks.workunit.client.0.smithi110.stderr:++ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:38:36.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ n=1 2024-05-04T17:38:36.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:38:36.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ striper rm baz.db 2024-05-04T17:38:36.984 INFO:tasks.workunit.client.0.smithi110.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-05-04T17:38:37.024 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:38:37.025 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:38:37 PM UTC 2024 2024-05-04T17:38:37.026 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite b 2024-05-04T17:38:37.026 INFO:tasks.workunit.client.0.smithi110.stderr:+ background=b 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' b = b ']' 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr:+ shift 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ cat 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr:+ a='CREATE TABLE foo (a BLOB); 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo VALUES ("start"); 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr:WITH RECURSIVE c(x) AS 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr: ( 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr: VALUES(1) 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr: UNION ALL 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT x+1 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:38:37.027 INFO:tasks.workunit.client.0.smithi110.stderr: ) 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo (a) 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT RANDOMBLOB(1<<20) 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: LIMIT (1<<20);' 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ printf %s 'CREATE TABLE foo (a BLOB); 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo VALUES ("start"); 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr:WITH RECURSIVE c(x) AS 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: ( 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: VALUES(1) 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: UNION ALL 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT x+1 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr: ) 2024-05-04T17:38:37.028 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo (a) 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT RANDOMBLOB(1<<20) 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: LIMIT (1<<20);' 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr:CREATE TABLE foo (a BLOB); 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo VALUES ("start"); 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr:WITH RECURSIVE c(x) AS 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: ( 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: VALUES(1) 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: UNION ALL 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT x+1 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr: ) 2024-05-04T17:38:37.029 INFO:tasks.workunit.client.0.smithi110.stderr:INSERT INTO foo (a) 2024-05-04T17:38:37.030 INFO:tasks.workunit.client.0.smithi110.stderr: SELECT RANDOMBLOB(1<<20) 2024-05-04T17:38:37.030 INFO:tasks.workunit.client.0.smithi110.stderr: FROM c 2024-05-04T17:38:37.030 INFO:tasks.workunit.client.0.smithi110.stderr: LIMIT (1<<20);+ '[' b '!=' b ']' 2024-05-04T17:38:37.030 INFO:tasks.workunit.client.0.smithi110.stderr:+ sleep 20 2024-05-04T17:38:37.030 INFO:tasks.workunit.client.0.smithi110.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:38:57.028 INFO:tasks.workunit.client.0.smithi110.stdout:[1]+ 36573 Running sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a" & 2024-05-04T17:38:57.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ jobs -l 2024-05-04T17:38:57.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ jobs -p 2024-05-04T17:38:57.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ kill -STOP -- 36573 2024-05-04T17:38:57.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ sleep 45 2024-05-04T17:39:42.030 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:39:42.031 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:39:42 PM UTC 2024 2024-05-04T17:39:42.031 INFO:tasks.workunit.client.0.smithi110.stderr:++ jobs -p 2024-05-04T17:39:42.032 INFO:tasks.workunit.client.0.smithi110.stderr:+ kill -CONT -- 36573 2024-05-04T17:39:42.032 INFO:tasks.workunit.client.0.smithi110.stderr:+ sleep 10 2024-05-04T17:39:42.053 INFO:tasks.workunit.client.0.smithi110.stderr:Error: near line 3: disk I/O error 2024-05-04T17:39:52.033 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:39:52.034 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:39:52 PM UTC 2024 2024-05-04T17:39:52.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 2024-05-04T17:39:52.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ date 2024-05-04T17:39:52.035 INFO:tasks.workunit.client.0.smithi110.stdout:Sat May 4 05:39:52 PM UTC 2024 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ sqlite 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ background= 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' = b ']' 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:+++ cat 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ a='SELECT COUNT(*) FROM foo;' 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %s 'SELECT COUNT(*) FROM foo;' 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:SELECT COUNT(*) FROM foo;++ '[' '' '!=' b ']' 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ wait 2024-05-04T17:39:52.036 INFO:tasks.workunit.client.0.smithi110.stderr:++ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-05-04T17:39:53.110 INFO:tasks.workunit.client.0.smithi110.stderr:+ n=1 2024-05-04T17:39:53.110 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:39:53.111 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-04T17:39:53.111 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-04T17:39:53.177 INFO:tasks.workunit:Stopping ['rados/test_libcephsqlite.sh cephsqlite'] on client.0... 2024-05-04T17:39:53.177 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-04T17:39:53.485 DEBUG:teuthology.parallel:result is None 2024-05-04T17:39:53.485 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-04T17:39:53.509 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-04T17:39:53.509 DEBUG:teuthology.orchestra.run.smithi110:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-04T17:39:53.570 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-04T17:39:53.570 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-04T17:39:53.585 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-04T17:39:53.585 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:39:53.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.705+0000 7f1ee2ad9640 1 Processor -- start 2024-05-04T17:39:53.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.705+0000 7f1ee2ad9640 1 -- start start 2024-05-04T17:39:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.705+0000 7f1ee2ad9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc072a20 con 0x7f1edc072640 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.705+0000 7f1ee2ad9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc13aa50 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.705+0000 7f1ee2ad9640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc140650 con 0x7f1edc1446c0 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1ee0875640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1edc072640 0x7f1edc13a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51874/0 (socket says 172.21.15.110:51874) 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1ee0875640 1 -- 172.21.15.110:0/3095725864 learned_addr learned my addr 172.21.15.110:0/3095725864 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262553733 0 0) 0x7f1edc072a20 con 0x7f1edc072640 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ecc003520 con 0x7f1edc072640 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804447887 0 0) 0x7f1edc140650 con 0x7f1edc1446c0 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1edc072a20 con 0x7f1edc1446c0 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 362003920 0 0) 0x7f1ecc003520 con 0x7f1edc072640 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 >> v1:172.21.15.110:6790/0 conn(0x7f1edc1446c0 legacy=0x7f1edc14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 >> v1:172.21.15.136:6789/0 conn(0x7f1edc0a9bd0 legacy=0x7f1edc143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc0aa2a0 con 0x7f1edc072640 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ed0001610 con 0x7f1edc072640 2024-05-04T17:39:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ed0001870 con 0x7f1edc072640 2024-05-04T17:39:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.706+0000 7f1edb7fe640 1 -- 172.21.15.110:0/3095725864 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ed0001c80 con 0x7f1edc072640 2024-05-04T17:39:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.707+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/3095725864 >> v1:172.21.15.110:6789/0 conn(0x7f1edc072640 legacy=0x7f1edc13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.707+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/3095725864 shutdown_connections 2024-05-04T17:39:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.707+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/3095725864 >> 172.21.15.110:0/3095725864 conn(0x7f1edc0b43a0 msgr2=0x7f1edc0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.707+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/3095725864 shutdown_connections 2024-05-04T17:39:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.707+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/3095725864 wait complete. 2024-05-04T17:39:53.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.708+0000 7f1ee2ad9640 1 Processor -- start 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.708+0000 7f1ee2ad9640 1 -- start start 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.708+0000 7f1ee2ad9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc0b02e0 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.708+0000 7f1ee2ad9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc13ed50 con 0x7f1edc1446c0 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.708+0000 7f1ee2ad9640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc070200 con 0x7f1edc072640 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ee0875640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f1edc072640 0x7f1edc0712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36698/0 (socket says 172.21.15.110:36698) 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ee0875640 1 -- 172.21.15.110:0/1770396026 learned_addr learned my addr 172.21.15.110:0/1770396026 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127029909 0 0) 0x7f1edc070200 con 0x7f1edc072640 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ec0003520 con 0x7f1edc072640 2024-05-04T17:39:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936677732 0 0) 0x7f1edc0b02e0 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1edc070200 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1710051879 0 0) 0x7f1ec0003520 con 0x7f1edc072640 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1edc0b02e0 con 0x7f1edc072640 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ed0001c80 con 0x7f1edc072640 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088030234 0 0) 0x7f1edc13ed50 con 0x7f1edc1446c0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ec0003520 con 0x7f1edc1446c0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3249778633 0 0) 0x7f1edc070200 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1edc13ed50 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ec8001830 con 0x7f1edc0a9bd0 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3113985773 0 0) 0x7f1edc0b02e0 con 0x7f1edc072640 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 >> v1:172.21.15.136:6789/0 conn(0x7f1edc1446c0 legacy=0x7f1edc13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 >> v1:172.21.15.110:6789/0 conn(0x7f1edc0a9bd0 legacy=0x7f1edc0afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc070700 con 0x7f1edc072640 2024-05-04T17:39:53.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc070930 con 0x7f1edc072640 2024-05-04T17:39:53.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1edc12d470 con 0x7f1edc072640 2024-05-04T17:39:53.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ed0001870 con 0x7f1edc072640 2024-05-04T17:39:53.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.709+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f1ed0004d90 con 0x7f1edc072640 2024-05-04T17:39:53.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.710+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ebc005080 con 0x7f1edc072640 2024-05-04T17:39:53.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.710+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f1ed0004fb0 con 0x7f1edc072640 2024-05-04T17:39:53.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.711+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f1ed00863f0 con 0x7f1edc072640 2024-05-04T17:39:53.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.715+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f1ed0055210 con 0x7f1edc072640 2024-05-04T17:39:53.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.885+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ebc000b90 con 0x7f1ec006c990 2024-05-04T17:39:53.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.888+0000 7f1ed97fa640 1 -- 172.21.15.110:0/1770396026 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117288 (unknown 2965378022 0 1030985541) 0x7f1ebc000b90 con 0x7f1ec006c990 2024-05-04T17:39:53.891 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:39:53.891 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:39:53.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.890+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 >> v1:172.21.15.110:6816/783516658 conn(0x7f1ec006c990 legacy=0x7f1ec006ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.890+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 >> v1:172.21.15.110:6790/0 conn(0x7f1edc072640 legacy=0x7f1edc0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:53.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.890+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 shutdown_connections 2024-05-04T17:39:53.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.890+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 >> 172.21.15.110:0/1770396026 conn(0x7f1edc0b43a0 msgr2=0x7f1edc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:53.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.890+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 shutdown_connections 2024-05-04T17:39:53.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:53.890+0000 7f1ee2ad9640 1 -- 172.21.15.110:0/1770396026 wait complete. 2024-05-04T17:39:53.903 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-05-04T17:39:53.173405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3579135272,"num_objects":883,"num_object_clones":0,"num_object_copies":1766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":2620,"num_read_kb":2615,"num_write":163773,"num_write_kb":5654609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166581,"ondisk_log_size":166581,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8442776,"kb_used_data":6701464,"kb_used_omap":12,"kb_used_meta":1741235,"kb_avail":746531944,"statfs":{"total":773094113280,"available":764448710656,"internally_reserved":0,"allocated":6862299136,"data_stored":6856905584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1783025201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002042"},"pg_stats":[{"pgid":"3.1f","version":"21'4742","reported_seq":4775,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922435+0000","last_change":"2024-05-04T17:36:05.257969+0000","last_active":"2024-05-04T17:38:36.922435+0000","last_peered":"2024-05-04T17:38:36.922435+0000","last_clean":"2024-05-04T17:38:36.922435+0000","last_became_active":"2024-05-04T17:36:05.257710+0000","last_became_peered":"2024-05-04T17:36:05.257710+0000","last_unstale":"2024-05-04T17:38:36.922435+0000","last_undegraded":"2024-05-04T17:38:36.922435+0000","last_fullsized":"2024-05-04T17:38:36.922435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:29:24.158209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":4760,"num_write_kb":219976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"22'6418","reported_seq":6441,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:43.256939+0000","last_change":"2024-05-04T17:36:05.257156+0000","last_active":"2024-05-04T17:38:43.256939+0000","last_peered":"2024-05-04T17:38:43.256939+0000","last_clean":"2024-05-04T17:38:43.256939+0000","last_became_active":"2024-05-04T17:36:05.256896+0000","last_became_peered":"2024-05-04T17:36:05.256896+0000","last_unstale":"2024-05-04T17:38:43.256939+0000","last_undegraded":"2024-05-04T17:38:43.256939+0000","last_fullsized":"2024-05-04T17:38:43.256939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6418,"log_dups_size":0,"ondisk_log_size":6418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:10:08.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377808,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":6410,"num_write_kb":223874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"22'5313","reported_seq":5338,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:48.287018+0000","last_change":"2024-05-04T17:36:05.260349+0000","last_active":"2024-05-04T17:38:48.287018+0000","last_peered":"2024-05-04T17:38:48.287018+0000","last_clean":"2024-05-04T17:38:48.287018+0000","last_became_active":"2024-05-04T17:36:05.260058+0000","last_became_peered":"2024-05-04T17:36:05.260058+0000","last_unstale":"2024-05-04T17:38:48.287018+0000","last_undegraded":"2024-05-04T17:38:48.287018+0000","last_fullsized":"2024-05-04T17:38:48.287018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:32:54.078676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121635328,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":5308,"num_write_kb":156688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"21'3541","reported_seq":3558,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.954480+0000","last_change":"2024-05-04T17:36:05.260218+0000","last_active":"2024-05-04T17:38:36.954480+0000","last_peered":"2024-05-04T17:38:36.954480+0000","last_clean":"2024-05-04T17:38:36.954480+0000","last_became_active":"2024-05-04T17:36:05.259933+0000","last_became_peered":"2024-05-04T17:36:05.259933+0000","last_unstale":"2024-05-04T17:38:36.954480+0000","last_undegraded":"2024-05-04T17:38:36.954480+0000","last_fullsized":"2024-05-04T17:38:36.954480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:56:35.210162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":168834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"22'3388","reported_seq":3405,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.151421+0000","last_change":"2024-05-04T17:36:05.260201+0000","last_active":"2024-05-04T17:38:37.151421+0000","last_peered":"2024-05-04T17:38:37.151421+0000","last_clean":"2024-05-04T17:38:37.151421+0000","last_became_active":"2024-05-04T17:36:05.259956+0000","last_became_peered":"2024-05-04T17:36:05.259956+0000","last_unstale":"2024-05-04T17:38:37.151421+0000","last_undegraded":"2024-05-04T17:38:37.151421+0000","last_fullsized":"2024-05-04T17:38:37.151421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:54:47.863593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":156550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"22'5220","reported_seq":5245,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:42.133922+0000","last_change":"2024-05-04T17:36:05.259945+0000","last_active":"2024-05-04T17:38:42.133922+0000","last_peered":"2024-05-04T17:38:42.133922+0000","last_clean":"2024-05-04T17:38:42.133922+0000","last_became_active":"2024-05-04T17:36:05.259594+0000","last_became_peered":"2024-05-04T17:36:05.259594+0000","last_unstale":"2024-05-04T17:38:42.133922+0000","last_undegraded":"2024-05-04T17:38:42.133922+0000","last_fullsized":"2024-05-04T17:38:42.133922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:00:55.808006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731664,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":28,"num_read_kb":21,"num_write":5212,"num_write_kb":167052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"22'4732","reported_seq":4749,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:52.840702+0000","last_change":"2024-05-04T17:36:05.250121+0000","last_active":"2024-05-04T17:38:52.840702+0000","last_peered":"2024-05-04T17:38:52.840702+0000","last_clean":"2024-05-04T17:38:52.840702+0000","last_became_active":"2024-05-04T17:36:05.249833+0000","last_became_peered":"2024-05-04T17:36:05.249833+0000","last_unstale":"2024-05-04T17:38:52.840702+0000","last_undegraded":"2024-05-04T17:38:52.840702+0000","last_fullsized":"2024-05-04T17:38:52.840702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:12:21.169603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4685,"num_write_kb":173061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"22'6599","reported_seq":6616,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:52.513832+0000","last_change":"2024-05-04T17:36:05.250305+0000","last_active":"2024-05-04T17:38:52.513832+0000","last_peered":"2024-05-04T17:38:52.513832+0000","last_clean":"2024-05-04T17:38:52.513832+0000","last_became_active":"2024-05-04T17:36:05.250054+0000","last_became_peered":"2024-05-04T17:36:05.250054+0000","last_unstale":"2024-05-04T17:38:52.513832+0000","last_undegraded":"2024-05-04T17:38:52.513832+0000","last_fullsized":"2024-05-04T17:38:52.513832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:20:34.169660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":157654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"22'7277","reported_seq":7297,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:55.970930+0000","last_change":"2024-05-04T17:36:05.258270+0000","last_active":"2024-05-04T17:38:55.970930+0000","last_peered":"2024-05-04T17:38:55.970930+0000","last_clean":"2024-05-04T17:38:55.970930+0000","last_became_active":"2024-05-04T17:36:05.257748+0000","last_became_peered":"2024-05-04T17:36:05.257748+0000","last_unstale":"2024-05-04T17:38:55.970930+0000","last_undegraded":"2024-05-04T17:38:55.970930+0000","last_fullsized":"2024-05-04T17:38:55.970930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:45:10.445360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":7280,"num_write_kb":228459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"22'5969","reported_seq":6296,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.098996+0000","last_change":"2024-05-04T17:36:05.264020+0000","last_active":"2024-05-04T17:38:37.098996+0000","last_peered":"2024-05-04T17:38:37.098996+0000","last_clean":"2024-05-04T17:38:37.098996+0000","last_became_active":"2024-05-04T17:36:05.263680+0000","last_became_peered":"2024-05-04T17:36:05.263680+0000","last_unstale":"2024-05-04T17:38:37.098996+0000","last_undegraded":"2024-05-04T17:38:37.098996+0000","last_fullsized":"2024-05-04T17:38:37.098996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5969,"log_dups_size":0,"ondisk_log_size":5969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:22:37.444606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126026248,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":1038,"num_read_kb":1054,"num_write":6173,"num_write_kb":205402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"22'8521","reported_seq":8554,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:51.899766+0000","last_change":"2024-05-04T17:36:05.260924+0000","last_active":"2024-05-04T17:38:51.899766+0000","last_peered":"2024-05-04T17:38:51.899766+0000","last_clean":"2024-05-04T17:38:51.899766+0000","last_became_active":"2024-05-04T17:36:05.260588+0000","last_became_peered":"2024-05-04T17:36:05.260588+0000","last_unstale":"2024-05-04T17:38:51.899766+0000","last_undegraded":"2024-05-04T17:38:51.899766+0000","last_fullsized":"2024-05-04T17:38:51.899766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:20:08.528563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":50,"num_read_kb":41,"num_write":8540,"num_write_kb":185557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"22'5980","reported_seq":6013,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:46.085231+0000","last_change":"2024-05-04T17:36:05.262544+0000","last_active":"2024-05-04T17:38:46.085231+0000","last_peered":"2024-05-04T17:38:46.085231+0000","last_clean":"2024-05-04T17:38:46.085231+0000","last_became_active":"2024-05-04T17:36:05.262061+0000","last_became_peered":"2024-05-04T17:36:05.262061+0000","last_unstale":"2024-05-04T17:38:46.085231+0000","last_undegraded":"2024-05-04T17:38:46.085231+0000","last_fullsized":"2024-05-04T17:38:46.085231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:50.820076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":5961,"num_write_kb":189699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.956384+0000","last_change":"2024-05-04T17:35:59.221521+0000","last_active":"2024-05-04T17:38:36.956384+0000","last_peered":"2024-05-04T17:38:36.956384+0000","last_clean":"2024-05-04T17:38:36.956384+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:38:36.956384+0000","last_undegraded":"2024-05-04T17:38:36.956384+0000","last_fullsized":"2024-05-04T17:38:36.956384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:11:32.394654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'6176","reported_seq":6197,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.054893+0000","last_change":"2024-05-04T17:36:05.247978+0000","last_active":"2024-05-04T17:39:42.054893+0000","last_peered":"2024-05-04T17:39:42.054893+0000","last_clean":"2024-05-04T17:39:42.054893+0000","last_became_active":"2024-05-04T17:36:05.247697+0000","last_became_peered":"2024-05-04T17:36:05.247697+0000","last_unstale":"2024-05-04T17:39:42.054893+0000","last_undegraded":"2024-05-04T17:39:42.054893+0000","last_fullsized":"2024-05-04T17:39:42.054893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:06:15.551179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78672920,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":6162,"num_write_kb":192039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.953151+0000","last_change":"2024-05-04T17:35:59.221461+0000","last_active":"2024-05-04T17:38:36.953151+0000","last_peered":"2024-05-04T17:38:36.953151+0000","last_clean":"2024-05-04T17:38:36.953151+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:38:36.953151+0000","last_undegraded":"2024-05-04T17:38:36.953151+0000","last_fullsized":"2024-05-04T17:38:36.953151+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:44:09.888594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'3984","reported_seq":4008,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.150017+0000","last_change":"2024-05-04T17:36:05.262127+0000","last_active":"2024-05-04T17:38:37.150017+0000","last_peered":"2024-05-04T17:38:37.150017+0000","last_clean":"2024-05-04T17:38:37.150017+0000","last_became_active":"2024-05-04T17:36:05.261828+0000","last_became_peered":"2024-05-04T17:36:05.261828+0000","last_unstale":"2024-05-04T17:38:37.150017+0000","last_undegraded":"2024-05-04T17:38:37.150017+0000","last_fullsized":"2024-05-04T17:38:37.150017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:57:34.308542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":3961,"num_write_kb":185542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929862+0000","last_change":"2024-05-04T17:35:59.222186+0000","last_active":"2024-05-04T17:38:36.929862+0000","last_peered":"2024-05-04T17:38:36.929862+0000","last_clean":"2024-05-04T17:38:36.929862+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:38:36.929862+0000","last_undegraded":"2024-05-04T17:38:36.929862+0000","last_fullsized":"2024-05-04T17:38:36.929862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:15:38.191128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'5132","reported_seq":5149,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:54.216371+0000","last_change":"2024-05-04T17:36:05.395570+0000","last_active":"2024-05-04T17:38:54.216371+0000","last_peered":"2024-05-04T17:38:54.216371+0000","last_clean":"2024-05-04T17:38:54.216371+0000","last_became_active":"2024-05-04T17:36:05.395267+0000","last_became_peered":"2024-05-04T17:36:05.395267+0000","last_unstale":"2024-05-04T17:38:54.216371+0000","last_undegraded":"2024-05-04T17:38:54.216371+0000","last_fullsized":"2024-05-04T17:38:54.216371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:48:40.897417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":103041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.952916+0000","last_change":"2024-05-04T17:35:59.218744+0000","last_active":"2024-05-04T17:38:36.952916+0000","last_peered":"2024-05-04T17:38:36.952916+0000","last_clean":"2024-05-04T17:38:36.952916+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:38:36.952916+0000","last_undegraded":"2024-05-04T17:38:36.952916+0000","last_fullsized":"2024-05-04T17:38:36.952916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:37:18.872091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'6471","reported_seq":6519,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.069551+0000","last_change":"2024-05-04T17:36:05.254630+0000","last_active":"2024-05-04T17:39:42.069551+0000","last_peered":"2024-05-04T17:39:42.069551+0000","last_clean":"2024-05-04T17:39:42.069551+0000","last_became_active":"2024-05-04T17:36:05.254049+0000","last_became_peered":"2024-05-04T17:36:05.254049+0000","last_unstale":"2024-05-04T17:39:42.069551+0000","last_undegraded":"2024-05-04T17:39:42.069551+0000","last_fullsized":"2024-05-04T17:39:42.069551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:59:05.699546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123508560,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":117,"num_read_kb":157,"num_write":6516,"num_write_kb":216324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'4380","reported_seq":4402,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922479+0000","last_change":"2024-05-04T17:36:05.260042+0000","last_active":"2024-05-04T17:38:36.922479+0000","last_peered":"2024-05-04T17:38:36.922479+0000","last_clean":"2024-05-04T17:38:36.922479+0000","last_became_active":"2024-05-04T17:36:05.259684+0000","last_became_peered":"2024-05-04T17:36:05.259684+0000","last_unstale":"2024-05-04T17:38:36.922479+0000","last_undegraded":"2024-05-04T17:38:36.922479+0000","last_fullsized":"2024-05-04T17:38:36.922479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:54.517545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829632,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":4394,"num_write_kb":197895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.955303+0000","last_change":"2024-05-04T17:35:59.223094+0000","last_active":"2024-05-04T17:38:36.955303+0000","last_peered":"2024-05-04T17:38:36.955303+0000","last_clean":"2024-05-04T17:38:36.955303+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:38:36.955303+0000","last_undegraded":"2024-05-04T17:38:36.955303+0000","last_fullsized":"2024-05-04T17:38:36.955303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:54:23.823020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"22'3964","reported_seq":3981,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.149971+0000","last_change":"2024-05-04T17:36:05.395580+0000","last_active":"2024-05-04T17:38:37.149971+0000","last_peered":"2024-05-04T17:38:37.149971+0000","last_clean":"2024-05-04T17:38:37.149971+0000","last_became_active":"2024-05-04T17:36:05.395272+0000","last_became_peered":"2024-05-04T17:36:05.395272+0000","last_unstale":"2024-05-04T17:38:37.149971+0000","last_undegraded":"2024-05-04T17:38:37.149971+0000","last_fullsized":"2024-05-04T17:38:37.149971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:23:05.867612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110366368,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":158385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.926727+0000","last_change":"2024-05-04T17:35:59.331384+0000","last_active":"2024-05-04T17:38:36.926727+0000","last_peered":"2024-05-04T17:38:36.926727+0000","last_clean":"2024-05-04T17:38:36.926727+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:38:36.926727+0000","last_undegraded":"2024-05-04T17:38:36.926727+0000","last_fullsized":"2024-05-04T17:38:36.926727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:24:22.601115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"22'4100","reported_seq":4117,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.099902+0000","last_change":"2024-05-04T17:36:05.261680+0000","last_active":"2024-05-04T17:38:37.099902+0000","last_peered":"2024-05-04T17:38:37.099902+0000","last_clean":"2024-05-04T17:38:37.099902+0000","last_became_active":"2024-05-04T17:36:05.261395+0000","last_became_peered":"2024-05-04T17:36:05.261395+0000","last_unstale":"2024-05-04T17:38:37.099902+0000","last_undegraded":"2024-05-04T17:38:37.099902+0000","last_fullsized":"2024-05-04T17:38:37.099902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:02:29.609251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106299392,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":186884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929479+0000","last_change":"2024-05-04T17:35:59.223322+0000","last_active":"2024-05-04T17:38:36.929479+0000","last_peered":"2024-05-04T17:38:36.929479+0000","last_clean":"2024-05-04T17:38:36.929479+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:38:36.929479+0000","last_undegraded":"2024-05-04T17:38:36.929479+0000","last_fullsized":"2024-05-04T17:38:36.929479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:34:10.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"22'5762","reported_seq":5779,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:50.022039+0000","last_change":"2024-05-04T17:36:05.258017+0000","last_active":"2024-05-04T17:38:50.022039+0000","last_peered":"2024-05-04T17:38:50.022039+0000","last_clean":"2024-05-04T17:38:50.022039+0000","last_became_active":"2024-05-04T17:36:05.257750+0000","last_became_peered":"2024-05-04T17:36:05.257750+0000","last_unstale":"2024-05-04T17:38:50.022039+0000","last_undegraded":"2024-05-04T17:38:50.022039+0000","last_fullsized":"2024-05-04T17:38:50.022039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:20:27.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":181199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.954319+0000","last_change":"2024-05-04T17:35:59.221969+0000","last_active":"2024-05-04T17:38:36.954319+0000","last_peered":"2024-05-04T17:38:36.954319+0000","last_clean":"2024-05-04T17:38:36.954319+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:38:36.954319+0000","last_undegraded":"2024-05-04T17:38:36.954319+0000","last_fullsized":"2024-05-04T17:38:36.954319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:22:19.190308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929485+0000","last_change":"2024-05-04T17:35:58.172775+0000","last_active":"2024-05-04T17:38:36.929485+0000","last_peered":"2024-05-04T17:38:36.929485+0000","last_clean":"2024-05-04T17:38:36.929485+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:38:36.929485+0000","last_undegraded":"2024-05-04T17:38:36.929485+0000","last_fullsized":"2024-05-04T17:38:36.929485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_clean_scrub_stamp":"2024-05-04T17:35:57.156954+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:43:38.793258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"22'2730","reported_seq":2758,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148498+0000","last_change":"2024-05-04T17:36:05.250179+0000","last_active":"2024-05-04T17:38:37.148498+0000","last_peered":"2024-05-04T17:38:37.148498+0000","last_clean":"2024-05-04T17:38:37.148498+0000","last_became_active":"2024-05-04T17:36:05.249926+0000","last_became_peered":"2024-05-04T17:36:05.249926+0000","last_unstale":"2024-05-04T17:38:37.148498+0000","last_undegraded":"2024-05-04T17:38:37.148498+0000","last_fullsized":"2024-05-04T17:38:37.148498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:51:15.283094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":2682,"num_write_kb":127745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"22'9936","reported_seq":10067,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:53.646563+0000","last_change":"2024-05-04T17:36:05.395659+0000","last_active":"2024-05-04T17:38:53.646563+0000","last_peered":"2024-05-04T17:38:53.646563+0000","last_clean":"2024-05-04T17:38:53.646563+0000","last_became_active":"2024-05-04T17:36:05.395416+0000","last_became_peered":"2024-05-04T17:36:05.395416+0000","last_unstale":"2024-05-04T17:38:53.646563+0000","last_undegraded":"2024-05-04T17:38:53.646563+0000","last_fullsized":"2024-05-04T17:38:53.646563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:18:49.259604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":361,"num_read_kb":322,"num_write":6850,"num_write_kb":187020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"22'4846","reported_seq":4866,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:54.291500+0000","last_change":"2024-05-04T17:36:05.257442+0000","last_active":"2024-05-04T17:38:54.291500+0000","last_peered":"2024-05-04T17:38:54.291500+0000","last_clean":"2024-05-04T17:38:54.291500+0000","last_became_active":"2024-05-04T17:36:05.257078+0000","last_became_peered":"2024-05-04T17:36:05.257078+0000","last_unstale":"2024-05-04T17:38:54.291500+0000","last_undegraded":"2024-05-04T17:38:54.291500+0000","last_fullsized":"2024-05-04T17:38:54.291500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:53:55.417005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4826,"num_write_kb":165126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"21'4340","reported_seq":4360,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922158+0000","last_change":"2024-05-04T17:36:05.250165+0000","last_active":"2024-05-04T17:38:36.922158+0000","last_peered":"2024-05-04T17:38:36.922158+0000","last_clean":"2024-05-04T17:38:36.922158+0000","last_became_active":"2024-05-04T17:36:05.249845+0000","last_became_peered":"2024-05-04T17:36:05.249845+0000","last_unstale":"2024-05-04T17:38:36.922158+0000","last_undegraded":"2024-05-04T17:38:36.922158+0000","last_fullsized":"2024-05-04T17:38:36.922158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:40:29.446049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":4449,"num_write_kb":193090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"22'5660","reported_seq":5802,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148711+0000","last_change":"2024-05-04T17:36:05.258056+0000","last_active":"2024-05-04T17:38:37.148711+0000","last_peered":"2024-05-04T17:38:37.148711+0000","last_clean":"2024-05-04T17:38:37.148711+0000","last_became_active":"2024-05-04T17:36:05.257760+0000","last_became_peered":"2024-05-04T17:36:05.257760+0000","last_unstale":"2024-05-04T17:38:37.148711+0000","last_undegraded":"2024-05-04T17:38:37.148711+0000","last_fullsized":"2024-05-04T17:38:37.148711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:21:44.622756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140053176,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":196,"num_read_kb":182,"num_write":5664,"num_write_kb":217362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"22'3531","reported_seq":3565,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.098371+0000","last_change":"2024-05-04T17:36:05.261546+0000","last_active":"2024-05-04T17:38:37.098371+0000","last_peered":"2024-05-04T17:38:37.098371+0000","last_clean":"2024-05-04T17:38:37.098371+0000","last_became_active":"2024-05-04T17:36:05.261259+0000","last_became_peered":"2024-05-04T17:36:05.261259+0000","last_unstale":"2024-05-04T17:38:37.098371+0000","last_undegraded":"2024-05-04T17:38:37.098371+0000","last_fullsized":"2024-05-04T17:38:37.098371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:56:45.912709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":66,"num_read_kb":115,"num_write":3541,"num_write_kb":181187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"22'4849","reported_seq":4934,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.189601+0000","last_change":"2024-05-04T17:36:05.263514+0000","last_active":"2024-05-04T17:38:37.189601+0000","last_peered":"2024-05-04T17:38:37.189601+0000","last_clean":"2024-05-04T17:38:37.189601+0000","last_became_active":"2024-05-04T17:36:05.263214+0000","last_became_peered":"2024-05-04T17:36:05.263214+0000","last_unstale":"2024-05-04T17:38:37.189601+0000","last_undegraded":"2024-05-04T17:38:37.189601+0000","last_fullsized":"2024-05-04T17:38:37.189601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:05.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121643536,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":259,"num_read_kb":194,"num_write":5097,"num_write_kb":162415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'3517","reported_seq":3534,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.056418+0000","last_change":"2024-05-04T17:36:05.263191+0000","last_active":"2024-05-04T17:39:42.056418+0000","last_peered":"2024-05-04T17:39:42.056418+0000","last_clean":"2024-05-04T17:39:42.056418+0000","last_became_active":"2024-05-04T17:36:05.262817+0000","last_became_peered":"2024-05-04T17:36:05.262817+0000","last_unstale":"2024-05-04T17:39:42.056418+0000","last_undegraded":"2024-05-04T17:39:42.056418+0000","last_fullsized":"2024-05-04T17:39:42.056418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:17:30.844586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80883712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":134410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"22'3809","reported_seq":3828,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148181+0000","last_change":"2024-05-04T17:36:05.250207+0000","last_active":"2024-05-04T17:38:37.148181+0000","last_peered":"2024-05-04T17:38:37.148181+0000","last_clean":"2024-05-04T17:38:37.148181+0000","last_became_active":"2024-05-04T17:36:05.249932+0000","last_became_peered":"2024-05-04T17:36:05.249932+0000","last_unstale":"2024-05-04T17:38:37.148181+0000","last_undegraded":"2024-05-04T17:38:37.148181+0000","last_fullsized":"2024-05-04T17:38:37.148181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:23:37.128989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112574992,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3768,"num_write_kb":163450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"21'3909","reported_seq":3933,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929503+0000","last_change":"2024-05-04T17:36:05.263401+0000","last_active":"2024-05-04T17:38:36.929503+0000","last_peered":"2024-05-04T17:38:36.929503+0000","last_clean":"2024-05-04T17:38:36.929503+0000","last_became_active":"2024-05-04T17:36:05.263000+0000","last_became_peered":"2024-05-04T17:36:05.263000+0000","last_unstale":"2024-05-04T17:38:36.929503+0000","last_undegraded":"2024-05-04T17:38:36.929503+0000","last_fullsized":"2024-05-04T17:38:36.929503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3909,"log_dups_size":0,"ondisk_log_size":3909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:51:16.339176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":3923,"num_write_kb":150485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"22'3427","reported_seq":3457,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.150540+0000","last_change":"2024-05-04T17:36:05.261174+0000","last_active":"2024-05-04T17:38:37.150540+0000","last_peered":"2024-05-04T17:38:37.150540+0000","last_clean":"2024-05-04T17:38:37.150540+0000","last_became_active":"2024-05-04T17:36:05.260881+0000","last_became_peered":"2024-05-04T17:36:05.260881+0000","last_unstale":"2024-05-04T17:38:37.150540+0000","last_undegraded":"2024-05-04T17:38:37.150540+0000","last_fullsized":"2024-05-04T17:38:37.150540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:28:46.613430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":54,"num_read_kb":41,"num_write":3408,"num_write_kb":117896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'8241","reported_seq":8261,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:48.823498+0000","last_change":"2024-05-04T17:36:05.263104+0000","last_active":"2024-05-04T17:38:48.823498+0000","last_peered":"2024-05-04T17:38:48.823498+0000","last_clean":"2024-05-04T17:38:48.823498+0000","last_became_active":"2024-05-04T17:36:05.262502+0000","last_became_peered":"2024-05-04T17:36:05.262502+0000","last_unstale":"2024-05-04T17:38:48.823498+0000","last_undegraded":"2024-05-04T17:38:48.823498+0000","last_fullsized":"2024-05-04T17:38:48.823498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:53:50.786398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8220,"num_write_kb":197892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3578544904,"num_objects":881,"num_object_clones":0,"num_object_copies":1762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":2504,"num_read_kb":2515,"num_write":163614,"num_write_kb":5652193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6859563008,"data_stored":6855288984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166464,"ondisk_log_size":166464,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542192,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343508,"kb_used_data":1055564,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93028332,"statfs":{"total":96636764160,"available":95261011968,"internally_reserved":0,"allocated":1080897536,"data_stored":1080011519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":294844874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542194,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266616,"kb_used_data":1005040,"kb_used_omap":1,"kb_used_meta":261566,"kb_avail":93105224,"statfs":{"total":96636764160,"available":95339749376,"internally_reserved":0,"allocated":1029160960,"data_stored":1028421251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267844037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542194,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1724732,"kb_used_data":1350772,"kb_used_omap":1,"kb_used_meta":373950,"kb_avail":92647108,"statfs":{"total":96636764160,"available":94870638592,"internally_reserved":0,"allocated":1383190528,"data_stored":1382090015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":382925253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401612,"kb_used_data":330244,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93970228,"statfs":{"total":96636764160,"available":96225513472,"internally_reserved":0,"allocated":338169856,"data_stored":337866195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362136,"kb_used_data":1074192,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93009704,"statfs":{"total":96636764160,"available":95241936896,"internally_reserved":0,"allocated":1099972608,"data_stored":1099151903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294844871},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377668,"kb_used_data":288188,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93994172,"statfs":{"total":96636764160,"available":96250032128,"internally_reserved":0,"allocated":295104512,"data_stored":294760883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649108,"kb_used_data":545804,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93722732,"statfs":{"total":96636764160,"available":95972077568,"internally_reserved":0,"allocated":558903296,"data_stored":558497191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1317396,"kb_used_data":1051660,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":93054444,"statfs":{"total":96636764160,"available":95287750656,"internally_reserved":0,"allocated":1076899840,"data_stored":1076106627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272103877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1076158464,"data_stored":1075507856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":558694400,"data_stored":558427348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294895616,"data_stored":294691040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1099763712,"data_stored":1099082060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1382981632,"data_stored":1382020172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1028952064,"data_stored":1028351408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":337960960,"data_stored":337796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080156160,"data_stored":1079412748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:39:53.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.003+0000 7fbcd5606640 1 Processor -- start 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.003+0000 7fbcd5606640 1 -- start start 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbcd5606640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd0146460 con 0x7fbcd0142340 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbcd5606640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd0073120 con 0x7fbcd0146710 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbcd5606640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd0071400 con 0x7fbcd0146080 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbcceffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fbcd0146080 0x7fbcd006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36704/0 (socket says 172.21.15.110:36704) 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbcceffd640 1 -- 172.21.15.110:0/3282094233 learned_addr learned my addr 172.21.15.110:0/3282094233 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198821776 0 0) 0x7fbcd0071400 con 0x7fbcd0146080 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb8003520 con 0x7fbcd0146080 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054279763 0 0) 0x7fbcd0146460 con 0x7fbcd0142340 2024-05-04T17:39:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd0071400 con 0x7fbcd0142340 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2196238855 0 0) 0x7fbcd0071400 con 0x7fbcd0142340 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 >> v1:172.21.15.110:6790/0 conn(0x7fbcd0146080 legacy=0x7fbcd006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 >> v1:172.21.15.136:6789/0 conn(0x7fbcd0146710 legacy=0x7fbcd0141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd0071f60 con 0x7fbcd0142340 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.004+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbcc8001610 con 0x7fbcd0142340 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbcc8001800 con 0x7fbcd0142340 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbccdffb640 1 -- 172.21.15.110:0/3282094233 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbcc8001c10 con 0x7fbcd0142340 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbcd5606640 1 -- 172.21.15.110:0/3282094233 >> v1:172.21.15.110:6789/0 conn(0x7fbcd0142340 legacy=0x7fbcd014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbcd5606640 1 -- 172.21.15.110:0/3282094233 shutdown_connections 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbcd5606640 1 -- 172.21.15.110:0/3282094233 >> 172.21.15.110:0/3282094233 conn(0x7fbcd00b8920 msgr2=0x7fbcd00b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbcd5606640 1 -- 172.21.15.110:0/3282094233 shutdown_connections 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.005+0000 7fbcd5606640 1 -- 172.21.15.110:0/3282094233 wait complete. 2024-05-04T17:39:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.006+0000 7fbcd5606640 1 Processor -- start 2024-05-04T17:39:54.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.006+0000 7fbcd5606640 1 -- start start 2024-05-04T17:39:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcd5606640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd01d87f0 con 0x7fbcd0142340 2024-05-04T17:39:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcd5606640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd00b4970 con 0x7fbcd0146710 2024-05-04T17:39:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcd5606640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd01db8f0 con 0x7fbcd0146080 2024-05-04T17:39:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcceffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbcd0142340 0x7fbcd00b0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51894/0 (socket says 172.21.15.110:51894) 2024-05-04T17:39:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcceffd640 1 -- 172.21.15.110:0/3677462212 learned_addr learned my addr 172.21.15.110:0/3677462212 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243031161 0 0) 0x7fbcd01db8f0 con 0x7fbcd0146080 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcbc003520 con 0x7fbcd0146080 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3601035641 0 0) 0x7fbcd01d87f0 con 0x7fbcd0142340 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd01db8f0 con 0x7fbcd0142340 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3208099897 0 0) 0x7fbcbc003520 con 0x7fbcd0146080 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcd01d87f0 con 0x7fbcd0146080 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbcc4001790 con 0x7fbcd0146080 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3403675149 0 0) 0x7fbcd01db8f0 con 0x7fbcd0142340 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcbc003520 con 0x7fbcd0142340 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbcc00016e0 con 0x7fbcd0142340 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640562618 0 0) 0x7fbcd00b4970 con 0x7fbcd0146710 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd01db8f0 con 0x7fbcd0146710 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3373532127 0 0) 0x7fbcd01d87f0 con 0x7fbcd0146080 2024-05-04T17:39:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 >> v1:172.21.15.136:6789/0 conn(0x7fbcd0146710 legacy=0x7fbcd013b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 >> v1:172.21.15.110:6789/0 conn(0x7fbcd0142340 legacy=0x7fbcd00b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.007+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd01dbac0 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.008+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd01d76a0 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.008+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbcd013c130 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.008+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbcc40035f0 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.008+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbcc4003940 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.009+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb0005080 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.009+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fbcc4003ae0 con 0x7fbcd0146080 2024-05-04T17:39:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.009+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fbcc4085080 con 0x7fbcd0146080 2024-05-04T17:39:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.013+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fbcc4053ea0 con 0x7fbcd0146080 2024-05-04T17:39:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.187+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbcb0000b90 con 0x7fbcbc06c990 2024-05-04T17:39:54.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.190+0000 7fbcb77fe640 1 -- 172.21.15.110:0/3677462212 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117288 (unknown 2965378022 0 1030985541) 0x7fbcb0000b90 con 0x7fbcbc06c990 2024-05-04T17:39:54.192 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:39:54.192 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:39:54.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.192+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 >> v1:172.21.15.110:6816/783516658 conn(0x7fbcbc06c990 legacy=0x7fbcbc06ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.192+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 >> v1:172.21.15.110:6790/0 conn(0x7fbcd0146080 legacy=0x7fbcd01d80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.192+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 shutdown_connections 2024-05-04T17:39:54.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.192+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 >> 172.21.15.110:0/3677462212 conn(0x7fbcd00b8920 msgr2=0x7fbcd0143110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:54.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.192+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 shutdown_connections 2024-05-04T17:39:54.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.192+0000 7fbcd5606640 1 -- 172.21.15.110:0/3677462212 wait complete. 2024-05-04T17:39:54.205 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-05-04T17:39:53.173405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3579135272,"num_objects":883,"num_object_clones":0,"num_object_copies":1766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":2620,"num_read_kb":2615,"num_write":163773,"num_write_kb":5654609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166581,"ondisk_log_size":166581,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8442776,"kb_used_data":6701464,"kb_used_omap":12,"kb_used_meta":1741235,"kb_avail":746531944,"statfs":{"total":773094113280,"available":764448710656,"internally_reserved":0,"allocated":6862299136,"data_stored":6856905584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1783025201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002042"},"pg_stats":[{"pgid":"3.1f","version":"21'4742","reported_seq":4775,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922435+0000","last_change":"2024-05-04T17:36:05.257969+0000","last_active":"2024-05-04T17:38:36.922435+0000","last_peered":"2024-05-04T17:38:36.922435+0000","last_clean":"2024-05-04T17:38:36.922435+0000","last_became_active":"2024-05-04T17:36:05.257710+0000","last_became_peered":"2024-05-04T17:36:05.257710+0000","last_unstale":"2024-05-04T17:38:36.922435+0000","last_undegraded":"2024-05-04T17:38:36.922435+0000","last_fullsized":"2024-05-04T17:38:36.922435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:29:24.158209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":4760,"num_write_kb":219976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"22'6418","reported_seq":6441,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:43.256939+0000","last_change":"2024-05-04T17:36:05.257156+0000","last_active":"2024-05-04T17:38:43.256939+0000","last_peered":"2024-05-04T17:38:43.256939+0000","last_clean":"2024-05-04T17:38:43.256939+0000","last_became_active":"2024-05-04T17:36:05.256896+0000","last_became_peered":"2024-05-04T17:36:05.256896+0000","last_unstale":"2024-05-04T17:38:43.256939+0000","last_undegraded":"2024-05-04T17:38:43.256939+0000","last_fullsized":"2024-05-04T17:38:43.256939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6418,"log_dups_size":0,"ondisk_log_size":6418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:10:08.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377808,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":6410,"num_write_kb":223874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"22'5313","reported_seq":5338,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:48.287018+0000","last_change":"2024-05-04T17:36:05.260349+0000","last_active":"2024-05-04T17:38:48.287018+0000","last_peered":"2024-05-04T17:38:48.287018+0000","last_clean":"2024-05-04T17:38:48.287018+0000","last_became_active":"2024-05-04T17:36:05.260058+0000","last_became_peered":"2024-05-04T17:36:05.260058+0000","last_unstale":"2024-05-04T17:38:48.287018+0000","last_undegraded":"2024-05-04T17:38:48.287018+0000","last_fullsized":"2024-05-04T17:38:48.287018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:32:54.078676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121635328,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":5308,"num_write_kb":156688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"21'3541","reported_seq":3558,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.954480+0000","last_change":"2024-05-04T17:36:05.260218+0000","last_active":"2024-05-04T17:38:36.954480+0000","last_peered":"2024-05-04T17:38:36.954480+0000","last_clean":"2024-05-04T17:38:36.954480+0000","last_became_active":"2024-05-04T17:36:05.259933+0000","last_became_peered":"2024-05-04T17:36:05.259933+0000","last_unstale":"2024-05-04T17:38:36.954480+0000","last_undegraded":"2024-05-04T17:38:36.954480+0000","last_fullsized":"2024-05-04T17:38:36.954480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:56:35.210162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":168834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"22'3388","reported_seq":3405,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.151421+0000","last_change":"2024-05-04T17:36:05.260201+0000","last_active":"2024-05-04T17:38:37.151421+0000","last_peered":"2024-05-04T17:38:37.151421+0000","last_clean":"2024-05-04T17:38:37.151421+0000","last_became_active":"2024-05-04T17:36:05.259956+0000","last_became_peered":"2024-05-04T17:36:05.259956+0000","last_unstale":"2024-05-04T17:38:37.151421+0000","last_undegraded":"2024-05-04T17:38:37.151421+0000","last_fullsized":"2024-05-04T17:38:37.151421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:54:47.863593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":156550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"22'5220","reported_seq":5245,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:42.133922+0000","last_change":"2024-05-04T17:36:05.259945+0000","last_active":"2024-05-04T17:38:42.133922+0000","last_peered":"2024-05-04T17:38:42.133922+0000","last_clean":"2024-05-04T17:38:42.133922+0000","last_became_active":"2024-05-04T17:36:05.259594+0000","last_became_peered":"2024-05-04T17:36:05.259594+0000","last_unstale":"2024-05-04T17:38:42.133922+0000","last_undegraded":"2024-05-04T17:38:42.133922+0000","last_fullsized":"2024-05-04T17:38:42.133922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:00:55.808006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731664,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":28,"num_read_kb":21,"num_write":5212,"num_write_kb":167052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"22'4732","reported_seq":4749,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:52.840702+0000","last_change":"2024-05-04T17:36:05.250121+0000","last_active":"2024-05-04T17:38:52.840702+0000","last_peered":"2024-05-04T17:38:52.840702+0000","last_clean":"2024-05-04T17:38:52.840702+0000","last_became_active":"2024-05-04T17:36:05.249833+0000","last_became_peered":"2024-05-04T17:36:05.249833+0000","last_unstale":"2024-05-04T17:38:52.840702+0000","last_undegraded":"2024-05-04T17:38:52.840702+0000","last_fullsized":"2024-05-04T17:38:52.840702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:12:21.169603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4685,"num_write_kb":173061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"22'6599","reported_seq":6616,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:52.513832+0000","last_change":"2024-05-04T17:36:05.250305+0000","last_active":"2024-05-04T17:38:52.513832+0000","last_peered":"2024-05-04T17:38:52.513832+0000","last_clean":"2024-05-04T17:38:52.513832+0000","last_became_active":"2024-05-04T17:36:05.250054+0000","last_became_peered":"2024-05-04T17:36:05.250054+0000","last_unstale":"2024-05-04T17:38:52.513832+0000","last_undegraded":"2024-05-04T17:38:52.513832+0000","last_fullsized":"2024-05-04T17:38:52.513832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:20:34.169660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":157654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"22'7277","reported_seq":7297,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:55.970930+0000","last_change":"2024-05-04T17:36:05.258270+0000","last_active":"2024-05-04T17:38:55.970930+0000","last_peered":"2024-05-04T17:38:55.970930+0000","last_clean":"2024-05-04T17:38:55.970930+0000","last_became_active":"2024-05-04T17:36:05.257748+0000","last_became_peered":"2024-05-04T17:36:05.257748+0000","last_unstale":"2024-05-04T17:38:55.970930+0000","last_undegraded":"2024-05-04T17:38:55.970930+0000","last_fullsized":"2024-05-04T17:38:55.970930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:45:10.445360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":7280,"num_write_kb":228459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"22'5969","reported_seq":6296,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.098996+0000","last_change":"2024-05-04T17:36:05.264020+0000","last_active":"2024-05-04T17:38:37.098996+0000","last_peered":"2024-05-04T17:38:37.098996+0000","last_clean":"2024-05-04T17:38:37.098996+0000","last_became_active":"2024-05-04T17:36:05.263680+0000","last_became_peered":"2024-05-04T17:36:05.263680+0000","last_unstale":"2024-05-04T17:38:37.098996+0000","last_undegraded":"2024-05-04T17:38:37.098996+0000","last_fullsized":"2024-05-04T17:38:37.098996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5969,"log_dups_size":0,"ondisk_log_size":5969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:22:37.444606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126026248,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":1038,"num_read_kb":1054,"num_write":6173,"num_write_kb":205402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"22'8521","reported_seq":8554,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:51.899766+0000","last_change":"2024-05-04T17:36:05.260924+0000","last_active":"2024-05-04T17:38:51.899766+0000","last_peered":"2024-05-04T17:38:51.899766+0000","last_clean":"2024-05-04T17:38:51.899766+0000","last_became_active":"2024-05-04T17:36:05.260588+0000","last_became_peered":"2024-05-04T17:36:05.260588+0000","last_unstale":"2024-05-04T17:38:51.899766+0000","last_undegraded":"2024-05-04T17:38:51.899766+0000","last_fullsized":"2024-05-04T17:38:51.899766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:20:08.528563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":50,"num_read_kb":41,"num_write":8540,"num_write_kb":185557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"22'5980","reported_seq":6013,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:46.085231+0000","last_change":"2024-05-04T17:36:05.262544+0000","last_active":"2024-05-04T17:38:46.085231+0000","last_peered":"2024-05-04T17:38:46.085231+0000","last_clean":"2024-05-04T17:38:46.085231+0000","last_became_active":"2024-05-04T17:36:05.262061+0000","last_became_peered":"2024-05-04T17:36:05.262061+0000","last_unstale":"2024-05-04T17:38:46.085231+0000","last_undegraded":"2024-05-04T17:38:46.085231+0000","last_fullsized":"2024-05-04T17:38:46.085231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:50.820076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":5961,"num_write_kb":189699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.956384+0000","last_change":"2024-05-04T17:35:59.221521+0000","last_active":"2024-05-04T17:38:36.956384+0000","last_peered":"2024-05-04T17:38:36.956384+0000","last_clean":"2024-05-04T17:38:36.956384+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:38:36.956384+0000","last_undegraded":"2024-05-04T17:38:36.956384+0000","last_fullsized":"2024-05-04T17:38:36.956384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:11:32.394654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'6176","reported_seq":6197,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.054893+0000","last_change":"2024-05-04T17:36:05.247978+0000","last_active":"2024-05-04T17:39:42.054893+0000","last_peered":"2024-05-04T17:39:42.054893+0000","last_clean":"2024-05-04T17:39:42.054893+0000","last_became_active":"2024-05-04T17:36:05.247697+0000","last_became_peered":"2024-05-04T17:36:05.247697+0000","last_unstale":"2024-05-04T17:39:42.054893+0000","last_undegraded":"2024-05-04T17:39:42.054893+0000","last_fullsized":"2024-05-04T17:39:42.054893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:06:15.551179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78672920,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":6162,"num_write_kb":192039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.953151+0000","last_change":"2024-05-04T17:35:59.221461+0000","last_active":"2024-05-04T17:38:36.953151+0000","last_peered":"2024-05-04T17:38:36.953151+0000","last_clean":"2024-05-04T17:38:36.953151+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:38:36.953151+0000","last_undegraded":"2024-05-04T17:38:36.953151+0000","last_fullsized":"2024-05-04T17:38:36.953151+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:44:09.888594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'3984","reported_seq":4008,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.150017+0000","last_change":"2024-05-04T17:36:05.262127+0000","last_active":"2024-05-04T17:38:37.150017+0000","last_peered":"2024-05-04T17:38:37.150017+0000","last_clean":"2024-05-04T17:38:37.150017+0000","last_became_active":"2024-05-04T17:36:05.261828+0000","last_became_peered":"2024-05-04T17:36:05.261828+0000","last_unstale":"2024-05-04T17:38:37.150017+0000","last_undegraded":"2024-05-04T17:38:37.150017+0000","last_fullsized":"2024-05-04T17:38:37.150017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:57:34.308542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":3961,"num_write_kb":185542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929862+0000","last_change":"2024-05-04T17:35:59.222186+0000","last_active":"2024-05-04T17:38:36.929862+0000","last_peered":"2024-05-04T17:38:36.929862+0000","last_clean":"2024-05-04T17:38:36.929862+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:38:36.929862+0000","last_undegraded":"2024-05-04T17:38:36.929862+0000","last_fullsized":"2024-05-04T17:38:36.929862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:15:38.191128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'5132","reported_seq":5149,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:54.216371+0000","last_change":"2024-05-04T17:36:05.395570+0000","last_active":"2024-05-04T17:38:54.216371+0000","last_peered":"2024-05-04T17:38:54.216371+0000","last_clean":"2024-05-04T17:38:54.216371+0000","last_became_active":"2024-05-04T17:36:05.395267+0000","last_became_peered":"2024-05-04T17:36:05.395267+0000","last_unstale":"2024-05-04T17:38:54.216371+0000","last_undegraded":"2024-05-04T17:38:54.216371+0000","last_fullsized":"2024-05-04T17:38:54.216371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:48:40.897417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":103041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.952916+0000","last_change":"2024-05-04T17:35:59.218744+0000","last_active":"2024-05-04T17:38:36.952916+0000","last_peered":"2024-05-04T17:38:36.952916+0000","last_clean":"2024-05-04T17:38:36.952916+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:38:36.952916+0000","last_undegraded":"2024-05-04T17:38:36.952916+0000","last_fullsized":"2024-05-04T17:38:36.952916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:37:18.872091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'6471","reported_seq":6519,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.069551+0000","last_change":"2024-05-04T17:36:05.254630+0000","last_active":"2024-05-04T17:39:42.069551+0000","last_peered":"2024-05-04T17:39:42.069551+0000","last_clean":"2024-05-04T17:39:42.069551+0000","last_became_active":"2024-05-04T17:36:05.254049+0000","last_became_peered":"2024-05-04T17:36:05.254049+0000","last_unstale":"2024-05-04T17:39:42.069551+0000","last_undegraded":"2024-05-04T17:39:42.069551+0000","last_fullsized":"2024-05-04T17:39:42.069551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:59:05.699546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123508560,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":117,"num_read_kb":157,"num_write":6516,"num_write_kb":216324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'4380","reported_seq":4402,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922479+0000","last_change":"2024-05-04T17:36:05.260042+0000","last_active":"2024-05-04T17:38:36.922479+0000","last_peered":"2024-05-04T17:38:36.922479+0000","last_clean":"2024-05-04T17:38:36.922479+0000","last_became_active":"2024-05-04T17:36:05.259684+0000","last_became_peered":"2024-05-04T17:36:05.259684+0000","last_unstale":"2024-05-04T17:38:36.922479+0000","last_undegraded":"2024-05-04T17:38:36.922479+0000","last_fullsized":"2024-05-04T17:38:36.922479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:54.517545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829632,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":4394,"num_write_kb":197895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.955303+0000","last_change":"2024-05-04T17:35:59.223094+0000","last_active":"2024-05-04T17:38:36.955303+0000","last_peered":"2024-05-04T17:38:36.955303+0000","last_clean":"2024-05-04T17:38:36.955303+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:38:36.955303+0000","last_undegraded":"2024-05-04T17:38:36.955303+0000","last_fullsized":"2024-05-04T17:38:36.955303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:54:23.823020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"22'3964","reported_seq":3981,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.149971+0000","last_change":"2024-05-04T17:36:05.395580+0000","last_active":"2024-05-04T17:38:37.149971+0000","last_peered":"2024-05-04T17:38:37.149971+0000","last_clean":"2024-05-04T17:38:37.149971+0000","last_became_active":"2024-05-04T17:36:05.395272+0000","last_became_peered":"2024-05-04T17:36:05.395272+0000","last_unstale":"2024-05-04T17:38:37.149971+0000","last_undegraded":"2024-05-04T17:38:37.149971+0000","last_fullsized":"2024-05-04T17:38:37.149971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:23:05.867612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110366368,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":158385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.926727+0000","last_change":"2024-05-04T17:35:59.331384+0000","last_active":"2024-05-04T17:38:36.926727+0000","last_peered":"2024-05-04T17:38:36.926727+0000","last_clean":"2024-05-04T17:38:36.926727+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:38:36.926727+0000","last_undegraded":"2024-05-04T17:38:36.926727+0000","last_fullsized":"2024-05-04T17:38:36.926727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:24:22.601115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"22'4100","reported_seq":4117,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.099902+0000","last_change":"2024-05-04T17:36:05.261680+0000","last_active":"2024-05-04T17:38:37.099902+0000","last_peered":"2024-05-04T17:38:37.099902+0000","last_clean":"2024-05-04T17:38:37.099902+0000","last_became_active":"2024-05-04T17:36:05.261395+0000","last_became_peered":"2024-05-04T17:36:05.261395+0000","last_unstale":"2024-05-04T17:38:37.099902+0000","last_undegraded":"2024-05-04T17:38:37.099902+0000","last_fullsized":"2024-05-04T17:38:37.099902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:02:29.609251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106299392,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":186884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929479+0000","last_change":"2024-05-04T17:35:59.223322+0000","last_active":"2024-05-04T17:38:36.929479+0000","last_peered":"2024-05-04T17:38:36.929479+0000","last_clean":"2024-05-04T17:38:36.929479+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:38:36.929479+0000","last_undegraded":"2024-05-04T17:38:36.929479+0000","last_fullsized":"2024-05-04T17:38:36.929479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:34:10.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"22'5762","reported_seq":5779,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:50.022039+0000","last_change":"2024-05-04T17:36:05.258017+0000","last_active":"2024-05-04T17:38:50.022039+0000","last_peered":"2024-05-04T17:38:50.022039+0000","last_clean":"2024-05-04T17:38:50.022039+0000","last_became_active":"2024-05-04T17:36:05.257750+0000","last_became_peered":"2024-05-04T17:36:05.257750+0000","last_unstale":"2024-05-04T17:38:50.022039+0000","last_undegraded":"2024-05-04T17:38:50.022039+0000","last_fullsized":"2024-05-04T17:38:50.022039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:20:27.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":181199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.954319+0000","last_change":"2024-05-04T17:35:59.221969+0000","last_active":"2024-05-04T17:38:36.954319+0000","last_peered":"2024-05-04T17:38:36.954319+0000","last_clean":"2024-05-04T17:38:36.954319+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:38:36.954319+0000","last_undegraded":"2024-05-04T17:38:36.954319+0000","last_fullsized":"2024-05-04T17:38:36.954319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:22:19.190308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929485+0000","last_change":"2024-05-04T17:35:58.172775+0000","last_active":"2024-05-04T17:38:36.929485+0000","last_peered":"2024-05-04T17:38:36.929485+0000","last_clean":"2024-05-04T17:38:36.929485+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:38:36.929485+0000","last_undegraded":"2024-05-04T17:38:36.929485+0000","last_fullsized":"2024-05-04T17:38:36.929485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_clean_scrub_stamp":"2024-05-04T17:35:57.156954+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:43:38.793258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"22'2730","reported_seq":2758,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148498+0000","last_change":"2024-05-04T17:36:05.250179+0000","last_active":"2024-05-04T17:38:37.148498+0000","last_peered":"2024-05-04T17:38:37.148498+0000","last_clean":"2024-05-04T17:38:37.148498+0000","last_became_active":"2024-05-04T17:36:05.249926+0000","last_became_peered":"2024-05-04T17:36:05.249926+0000","last_unstale":"2024-05-04T17:38:37.148498+0000","last_undegraded":"2024-05-04T17:38:37.148498+0000","last_fullsized":"2024-05-04T17:38:37.148498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:51:15.283094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":2682,"num_write_kb":127745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"22'9936","reported_seq":10067,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:53.646563+0000","last_change":"2024-05-04T17:36:05.395659+0000","last_active":"2024-05-04T17:38:53.646563+0000","last_peered":"2024-05-04T17:38:53.646563+0000","last_clean":"2024-05-04T17:38:53.646563+0000","last_became_active":"2024-05-04T17:36:05.395416+0000","last_became_peered":"2024-05-04T17:36:05.395416+0000","last_unstale":"2024-05-04T17:38:53.646563+0000","last_undegraded":"2024-05-04T17:38:53.646563+0000","last_fullsized":"2024-05-04T17:38:53.646563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:18:49.259604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":361,"num_read_kb":322,"num_write":6850,"num_write_kb":187020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"22'4846","reported_seq":4866,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:54.291500+0000","last_change":"2024-05-04T17:36:05.257442+0000","last_active":"2024-05-04T17:38:54.291500+0000","last_peered":"2024-05-04T17:38:54.291500+0000","last_clean":"2024-05-04T17:38:54.291500+0000","last_became_active":"2024-05-04T17:36:05.257078+0000","last_became_peered":"2024-05-04T17:36:05.257078+0000","last_unstale":"2024-05-04T17:38:54.291500+0000","last_undegraded":"2024-05-04T17:38:54.291500+0000","last_fullsized":"2024-05-04T17:38:54.291500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:53:55.417005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4826,"num_write_kb":165126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"21'4340","reported_seq":4360,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922158+0000","last_change":"2024-05-04T17:36:05.250165+0000","last_active":"2024-05-04T17:38:36.922158+0000","last_peered":"2024-05-04T17:38:36.922158+0000","last_clean":"2024-05-04T17:38:36.922158+0000","last_became_active":"2024-05-04T17:36:05.249845+0000","last_became_peered":"2024-05-04T17:36:05.249845+0000","last_unstale":"2024-05-04T17:38:36.922158+0000","last_undegraded":"2024-05-04T17:38:36.922158+0000","last_fullsized":"2024-05-04T17:38:36.922158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:40:29.446049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":4449,"num_write_kb":193090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"22'5660","reported_seq":5802,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148711+0000","last_change":"2024-05-04T17:36:05.258056+0000","last_active":"2024-05-04T17:38:37.148711+0000","last_peered":"2024-05-04T17:38:37.148711+0000","last_clean":"2024-05-04T17:38:37.148711+0000","last_became_active":"2024-05-04T17:36:05.257760+0000","last_became_peered":"2024-05-04T17:36:05.257760+0000","last_unstale":"2024-05-04T17:38:37.148711+0000","last_undegraded":"2024-05-04T17:38:37.148711+0000","last_fullsized":"2024-05-04T17:38:37.148711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:21:44.622756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140053176,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":196,"num_read_kb":182,"num_write":5664,"num_write_kb":217362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"22'3531","reported_seq":3565,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.098371+0000","last_change":"2024-05-04T17:36:05.261546+0000","last_active":"2024-05-04T17:38:37.098371+0000","last_peered":"2024-05-04T17:38:37.098371+0000","last_clean":"2024-05-04T17:38:37.098371+0000","last_became_active":"2024-05-04T17:36:05.261259+0000","last_became_peered":"2024-05-04T17:36:05.261259+0000","last_unstale":"2024-05-04T17:38:37.098371+0000","last_undegraded":"2024-05-04T17:38:37.098371+0000","last_fullsized":"2024-05-04T17:38:37.098371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:56:45.912709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":66,"num_read_kb":115,"num_write":3541,"num_write_kb":181187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"22'4849","reported_seq":4934,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.189601+0000","last_change":"2024-05-04T17:36:05.263514+0000","last_active":"2024-05-04T17:38:37.189601+0000","last_peered":"2024-05-04T17:38:37.189601+0000","last_clean":"2024-05-04T17:38:37.189601+0000","last_became_active":"2024-05-04T17:36:05.263214+0000","last_became_peered":"2024-05-04T17:36:05.263214+0000","last_unstale":"2024-05-04T17:38:37.189601+0000","last_undegraded":"2024-05-04T17:38:37.189601+0000","last_fullsized":"2024-05-04T17:38:37.189601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:05.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121643536,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":259,"num_read_kb":194,"num_write":5097,"num_write_kb":162415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'3517","reported_seq":3534,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.056418+0000","last_change":"2024-05-04T17:36:05.263191+0000","last_active":"2024-05-04T17:39:42.056418+0000","last_peered":"2024-05-04T17:39:42.056418+0000","last_clean":"2024-05-04T17:39:42.056418+0000","last_became_active":"2024-05-04T17:36:05.262817+0000","last_became_peered":"2024-05-04T17:36:05.262817+0000","last_unstale":"2024-05-04T17:39:42.056418+0000","last_undegraded":"2024-05-04T17:39:42.056418+0000","last_fullsized":"2024-05-04T17:39:42.056418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:17:30.844586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80883712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":134410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"22'3809","reported_seq":3828,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148181+0000","last_change":"2024-05-04T17:36:05.250207+0000","last_active":"2024-05-04T17:38:37.148181+0000","last_peered":"2024-05-04T17:38:37.148181+0000","last_clean":"2024-05-04T17:38:37.148181+0000","last_became_active":"2024-05-04T17:36:05.249932+0000","last_became_peered":"2024-05-04T17:36:05.249932+0000","last_unstale":"2024-05-04T17:38:37.148181+0000","last_undegraded":"2024-05-04T17:38:37.148181+0000","last_fullsized":"2024-05-04T17:38:37.148181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:23:37.128989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112574992,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3768,"num_write_kb":163450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"21'3909","reported_seq":3933,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929503+0000","last_change":"2024-05-04T17:36:05.263401+0000","last_active":"2024-05-04T17:38:36.929503+0000","last_peered":"2024-05-04T17:38:36.929503+0000","last_clean":"2024-05-04T17:38:36.929503+0000","last_became_active":"2024-05-04T17:36:05.263000+0000","last_became_peered":"2024-05-04T17:36:05.263000+0000","last_unstale":"2024-05-04T17:38:36.929503+0000","last_undegraded":"2024-05-04T17:38:36.929503+0000","last_fullsized":"2024-05-04T17:38:36.929503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3909,"log_dups_size":0,"ondisk_log_size":3909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:51:16.339176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":3923,"num_write_kb":150485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"22'3427","reported_seq":3457,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.150540+0000","last_change":"2024-05-04T17:36:05.261174+0000","last_active":"2024-05-04T17:38:37.150540+0000","last_peered":"2024-05-04T17:38:37.150540+0000","last_clean":"2024-05-04T17:38:37.150540+0000","last_became_active":"2024-05-04T17:36:05.260881+0000","last_became_peered":"2024-05-04T17:36:05.260881+0000","last_unstale":"2024-05-04T17:38:37.150540+0000","last_undegraded":"2024-05-04T17:38:37.150540+0000","last_fullsized":"2024-05-04T17:38:37.150540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:28:46.613430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":54,"num_read_kb":41,"num_write":3408,"num_write_kb":117896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'8241","reported_seq":8261,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:48.823498+0000","last_change":"2024-05-04T17:36:05.263104+0000","last_active":"2024-05-04T17:38:48.823498+0000","last_peered":"2024-05-04T17:38:48.823498+0000","last_clean":"2024-05-04T17:38:48.823498+0000","last_became_active":"2024-05-04T17:36:05.262502+0000","last_became_peered":"2024-05-04T17:36:05.262502+0000","last_unstale":"2024-05-04T17:38:48.823498+0000","last_undegraded":"2024-05-04T17:38:48.823498+0000","last_fullsized":"2024-05-04T17:38:48.823498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:53:50.786398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8220,"num_write_kb":197892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3578544904,"num_objects":881,"num_object_clones":0,"num_object_copies":1762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":2504,"num_read_kb":2515,"num_write":163614,"num_write_kb":5652193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6859563008,"data_stored":6855288984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166464,"ondisk_log_size":166464,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542192,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343508,"kb_used_data":1055564,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93028332,"statfs":{"total":96636764160,"available":95261011968,"internally_reserved":0,"allocated":1080897536,"data_stored":1080011519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":294844874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542194,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266616,"kb_used_data":1005040,"kb_used_omap":1,"kb_used_meta":261566,"kb_avail":93105224,"statfs":{"total":96636764160,"available":95339749376,"internally_reserved":0,"allocated":1029160960,"data_stored":1028421251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267844037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542194,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1724732,"kb_used_data":1350772,"kb_used_omap":1,"kb_used_meta":373950,"kb_avail":92647108,"statfs":{"total":96636764160,"available":94870638592,"internally_reserved":0,"allocated":1383190528,"data_stored":1382090015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":382925253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401612,"kb_used_data":330244,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93970228,"statfs":{"total":96636764160,"available":96225513472,"internally_reserved":0,"allocated":338169856,"data_stored":337866195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362136,"kb_used_data":1074192,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93009704,"statfs":{"total":96636764160,"available":95241936896,"internally_reserved":0,"allocated":1099972608,"data_stored":1099151903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294844871},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377668,"kb_used_data":288188,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93994172,"statfs":{"total":96636764160,"available":96250032128,"internally_reserved":0,"allocated":295104512,"data_stored":294760883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649108,"kb_used_data":545804,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93722732,"statfs":{"total":96636764160,"available":95972077568,"internally_reserved":0,"allocated":558903296,"data_stored":558497191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1317396,"kb_used_data":1051660,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":93054444,"statfs":{"total":96636764160,"available":95287750656,"internally_reserved":0,"allocated":1076899840,"data_stored":1076106627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272103877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1076158464,"data_stored":1075507856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":558694400,"data_stored":558427348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294895616,"data_stored":294691040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1099763712,"data_stored":1099082060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1382981632,"data_stored":1382020172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1028952064,"data_stored":1028351408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":337960960,"data_stored":337796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080156160,"data_stored":1079412748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:39:54.213 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-04T17:39:54.213 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:39:54.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.309+0000 7fdf2c7ab640 1 Processor -- start 2024-05-04T17:39:54.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.309+0000 7fdf2c7ab640 1 -- start start 2024-05-04T17:39:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.310+0000 7fdf2c7ab640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf2413fda0 con 0x7fdf24142480 2024-05-04T17:39:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.310+0000 7fdf2c7ab640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf240715c0 con 0x7fdf2406f830 2024-05-04T17:39:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.310+0000 7fdf2c7ab640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf2413e900 con 0x7fdf24142f50 2024-05-04T17:39:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.310+0000 7fdf2a547640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fdf24142f50 0x7fdf2413e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36716/0 (socket says 172.21.15.110:36716) 2024-05-04T17:39:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.310+0000 7fdf2ad48640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdf24142480 0x7fdf2414acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51896/0 (socket says 172.21.15.110:51896) 2024-05-04T17:39:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.310+0000 7fdf2a547640 1 -- 172.21.15.110:0/559408569 learned_addr learned my addr 172.21.15.110:0/559408569 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975023993 0 0) 0x7fdf2413e900 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf10003520 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2587180052 0 0) 0x7fdf2413fda0 con 0x7fdf24142480 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf2413e900 con 0x7fdf24142480 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1740801487 0 0) 0x7fdf10003520 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 >> v1:172.21.15.136:6789/0 conn(0x7fdf2406f830 legacy=0x7fdf24147590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 >> v1:172.21.15.110:6789/0 conn(0x7fdf24142480 legacy=0x7fdf2414acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf24072660 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdf20001610 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf20001870 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf29545640 1 -- 172.21.15.110:0/559408569 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdf20001c40 con 0x7fdf24142f50 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.311+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/559408569 >> v1:172.21.15.110:6790/0 conn(0x7fdf24142f50 legacy=0x7fdf2413e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.312+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/559408569 shutdown_connections 2024-05-04T17:39:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.312+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/559408569 >> 172.21.15.110:0/559408569 conn(0x7fdf240b43a0 msgr2=0x7fdf240b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:54.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.312+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/559408569 shutdown_connections 2024-05-04T17:39:54.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.312+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/559408569 wait complete. 2024-05-04T17:39:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2c7ab640 1 Processor -- start 2024-05-04T17:39:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2c7ab640 1 -- start start 2024-05-04T17:39:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2c7ab640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf2413b0f0 con 0x7fdf2406f830 2024-05-04T17:39:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2c7ab640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf240ab110 con 0x7fdf241d8720 2024-05-04T17:39:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2c7ab640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf2413d790 con 0x7fdf24142480 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2a547640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdf2406f830 0x7fdf240af920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51898/0 (socket says 172.21.15.110:51898) 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf2a547640 1 -- 172.21.15.110:0/1039973819 learned_addr learned my addr 172.21.15.110:0/1039973819 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf29d46640 1 -- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6790/0 conn(0x7fdf24142480 legacy=0x7fdf2413b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf29d46640 1 -- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6790/0 conn(0x7fdf24142480 legacy=0x7fdf2413b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf29d46640 1 --1- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6790/0 conn(0x7fdf24142480 0x7fdf2413b610 :-1 s=OPENED pgs=283 cs=1 l=1).handle_message read tag failed 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf29d46640 1 --1- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6790/0 conn(0x7fdf24142480 0x7fdf2413b610 :-1 s=OPENED pgs=283 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338698365 0 0) 0x7fdf2413b0f0 con 0x7fdf2406f830 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.313+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf08003520 con 0x7fdf2406f830 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526033182 0 0) 0x7fdf240ab110 con 0x7fdf241d8720 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf2413b0f0 con 0x7fdf241d8720 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2093683471 0 0) 0x7fdf08003520 con 0x7fdf2406f830 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf240ab110 con 0x7fdf2406f830 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdf20001510 con 0x7fdf2406f830 2024-05-04T17:39:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 664863819 0 0) 0x7fdf240ab110 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6790/0 conn(0x7fdf24142480 legacy=0x7fdf2413b610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 >> v1:172.21.15.136:6789/0 conn(0x7fdf241d8720 legacy=0x7fdf2413bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2413acd0 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf241d7c90 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf240b0030 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf20001870 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.314+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fdf20004a90 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.315+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf0c005080 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.315+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fdf2001ae20 con 0x7fdf2406f830 2024-05-04T17:39:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.316+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fdf20086200 con 0x7fdf2406f830 2024-05-04T17:39:54.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.320+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fdf20055020 con 0x7fdf2406f830 2024-05-04T17:39:54.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.491+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf0c000b90 con 0x7fdf0806ca00 2024-05-04T17:39:54.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.494+0000 7fdf16ffd640 1 -- 172.21.15.110:0/1039973819 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117288 (unknown 2965378022 0 1030985541) 0x7fdf0c000b90 con 0x7fdf0806ca00 2024-05-04T17:39:54.496 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:39:54.497 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:39:54.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.496+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6816/783516658 conn(0x7fdf0806ca00 legacy=0x7fdf0806eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.496+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 >> v1:172.21.15.110:6789/0 conn(0x7fdf2406f830 legacy=0x7fdf240af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.496+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 shutdown_connections 2024-05-04T17:39:54.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.496+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 >> 172.21.15.110:0/1039973819 conn(0x7fdf240b43a0 msgr2=0x7fdf2413ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:54.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.496+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 shutdown_connections 2024-05-04T17:39:54.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.496+0000 7fdf2c7ab640 1 -- 172.21.15.110:0/1039973819 wait complete. 2024-05-04T17:39:54.508 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-05-04T17:39:53.173405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3579135272,"num_objects":883,"num_object_clones":0,"num_object_copies":1766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":2620,"num_read_kb":2615,"num_write":163773,"num_write_kb":5654609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166581,"ondisk_log_size":166581,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8442776,"kb_used_data":6701464,"kb_used_omap":12,"kb_used_meta":1741235,"kb_avail":746531944,"statfs":{"total":773094113280,"available":764448710656,"internally_reserved":0,"allocated":6862299136,"data_stored":6856905584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1783025201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002042"},"pg_stats":[{"pgid":"3.1f","version":"21'4742","reported_seq":4775,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922435+0000","last_change":"2024-05-04T17:36:05.257969+0000","last_active":"2024-05-04T17:38:36.922435+0000","last_peered":"2024-05-04T17:38:36.922435+0000","last_clean":"2024-05-04T17:38:36.922435+0000","last_became_active":"2024-05-04T17:36:05.257710+0000","last_became_peered":"2024-05-04T17:36:05.257710+0000","last_unstale":"2024-05-04T17:38:36.922435+0000","last_undegraded":"2024-05-04T17:38:36.922435+0000","last_fullsized":"2024-05-04T17:38:36.922435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:29:24.158209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":4760,"num_write_kb":219976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"22'6418","reported_seq":6441,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:43.256939+0000","last_change":"2024-05-04T17:36:05.257156+0000","last_active":"2024-05-04T17:38:43.256939+0000","last_peered":"2024-05-04T17:38:43.256939+0000","last_clean":"2024-05-04T17:38:43.256939+0000","last_became_active":"2024-05-04T17:36:05.256896+0000","last_became_peered":"2024-05-04T17:36:05.256896+0000","last_unstale":"2024-05-04T17:38:43.256939+0000","last_undegraded":"2024-05-04T17:38:43.256939+0000","last_fullsized":"2024-05-04T17:38:43.256939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6418,"log_dups_size":0,"ondisk_log_size":6418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:10:08.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377808,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":6410,"num_write_kb":223874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"22'5313","reported_seq":5338,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:48.287018+0000","last_change":"2024-05-04T17:36:05.260349+0000","last_active":"2024-05-04T17:38:48.287018+0000","last_peered":"2024-05-04T17:38:48.287018+0000","last_clean":"2024-05-04T17:38:48.287018+0000","last_became_active":"2024-05-04T17:36:05.260058+0000","last_became_peered":"2024-05-04T17:36:05.260058+0000","last_unstale":"2024-05-04T17:38:48.287018+0000","last_undegraded":"2024-05-04T17:38:48.287018+0000","last_fullsized":"2024-05-04T17:38:48.287018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:32:54.078676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121635328,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":5308,"num_write_kb":156688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"21'3541","reported_seq":3558,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.954480+0000","last_change":"2024-05-04T17:36:05.260218+0000","last_active":"2024-05-04T17:38:36.954480+0000","last_peered":"2024-05-04T17:38:36.954480+0000","last_clean":"2024-05-04T17:38:36.954480+0000","last_became_active":"2024-05-04T17:36:05.259933+0000","last_became_peered":"2024-05-04T17:36:05.259933+0000","last_unstale":"2024-05-04T17:38:36.954480+0000","last_undegraded":"2024-05-04T17:38:36.954480+0000","last_fullsized":"2024-05-04T17:38:36.954480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:56:35.210162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":168834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"22'3388","reported_seq":3405,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.151421+0000","last_change":"2024-05-04T17:36:05.260201+0000","last_active":"2024-05-04T17:38:37.151421+0000","last_peered":"2024-05-04T17:38:37.151421+0000","last_clean":"2024-05-04T17:38:37.151421+0000","last_became_active":"2024-05-04T17:36:05.259956+0000","last_became_peered":"2024-05-04T17:36:05.259956+0000","last_unstale":"2024-05-04T17:38:37.151421+0000","last_undegraded":"2024-05-04T17:38:37.151421+0000","last_fullsized":"2024-05-04T17:38:37.151421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:54:47.863593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":156550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"22'5220","reported_seq":5245,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:42.133922+0000","last_change":"2024-05-04T17:36:05.259945+0000","last_active":"2024-05-04T17:38:42.133922+0000","last_peered":"2024-05-04T17:38:42.133922+0000","last_clean":"2024-05-04T17:38:42.133922+0000","last_became_active":"2024-05-04T17:36:05.259594+0000","last_became_peered":"2024-05-04T17:36:05.259594+0000","last_unstale":"2024-05-04T17:38:42.133922+0000","last_undegraded":"2024-05-04T17:38:42.133922+0000","last_fullsized":"2024-05-04T17:38:42.133922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:00:55.808006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731664,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":28,"num_read_kb":21,"num_write":5212,"num_write_kb":167052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"22'4732","reported_seq":4749,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:52.840702+0000","last_change":"2024-05-04T17:36:05.250121+0000","last_active":"2024-05-04T17:38:52.840702+0000","last_peered":"2024-05-04T17:38:52.840702+0000","last_clean":"2024-05-04T17:38:52.840702+0000","last_became_active":"2024-05-04T17:36:05.249833+0000","last_became_peered":"2024-05-04T17:36:05.249833+0000","last_unstale":"2024-05-04T17:38:52.840702+0000","last_undegraded":"2024-05-04T17:38:52.840702+0000","last_fullsized":"2024-05-04T17:38:52.840702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:12:21.169603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4685,"num_write_kb":173061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"22'6599","reported_seq":6616,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:52.513832+0000","last_change":"2024-05-04T17:36:05.250305+0000","last_active":"2024-05-04T17:38:52.513832+0000","last_peered":"2024-05-04T17:38:52.513832+0000","last_clean":"2024-05-04T17:38:52.513832+0000","last_became_active":"2024-05-04T17:36:05.250054+0000","last_became_peered":"2024-05-04T17:36:05.250054+0000","last_unstale":"2024-05-04T17:38:52.513832+0000","last_undegraded":"2024-05-04T17:38:52.513832+0000","last_fullsized":"2024-05-04T17:38:52.513832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:20:34.169660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":157654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"22'7277","reported_seq":7297,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:55.970930+0000","last_change":"2024-05-04T17:36:05.258270+0000","last_active":"2024-05-04T17:38:55.970930+0000","last_peered":"2024-05-04T17:38:55.970930+0000","last_clean":"2024-05-04T17:38:55.970930+0000","last_became_active":"2024-05-04T17:36:05.257748+0000","last_became_peered":"2024-05-04T17:36:05.257748+0000","last_unstale":"2024-05-04T17:38:55.970930+0000","last_undegraded":"2024-05-04T17:38:55.970930+0000","last_fullsized":"2024-05-04T17:38:55.970930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:45:10.445360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":7280,"num_write_kb":228459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"22'5969","reported_seq":6296,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.098996+0000","last_change":"2024-05-04T17:36:05.264020+0000","last_active":"2024-05-04T17:38:37.098996+0000","last_peered":"2024-05-04T17:38:37.098996+0000","last_clean":"2024-05-04T17:38:37.098996+0000","last_became_active":"2024-05-04T17:36:05.263680+0000","last_became_peered":"2024-05-04T17:36:05.263680+0000","last_unstale":"2024-05-04T17:38:37.098996+0000","last_undegraded":"2024-05-04T17:38:37.098996+0000","last_fullsized":"2024-05-04T17:38:37.098996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5969,"log_dups_size":0,"ondisk_log_size":5969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:22:37.444606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126026248,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":1038,"num_read_kb":1054,"num_write":6173,"num_write_kb":205402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"22'8521","reported_seq":8554,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:51.899766+0000","last_change":"2024-05-04T17:36:05.260924+0000","last_active":"2024-05-04T17:38:51.899766+0000","last_peered":"2024-05-04T17:38:51.899766+0000","last_clean":"2024-05-04T17:38:51.899766+0000","last_became_active":"2024-05-04T17:36:05.260588+0000","last_became_peered":"2024-05-04T17:36:05.260588+0000","last_unstale":"2024-05-04T17:38:51.899766+0000","last_undegraded":"2024-05-04T17:38:51.899766+0000","last_fullsized":"2024-05-04T17:38:51.899766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:20:08.528563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":50,"num_read_kb":41,"num_write":8540,"num_write_kb":185557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"22'5980","reported_seq":6013,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:46.085231+0000","last_change":"2024-05-04T17:36:05.262544+0000","last_active":"2024-05-04T17:38:46.085231+0000","last_peered":"2024-05-04T17:38:46.085231+0000","last_clean":"2024-05-04T17:38:46.085231+0000","last_became_active":"2024-05-04T17:36:05.262061+0000","last_became_peered":"2024-05-04T17:36:05.262061+0000","last_unstale":"2024-05-04T17:38:46.085231+0000","last_undegraded":"2024-05-04T17:38:46.085231+0000","last_fullsized":"2024-05-04T17:38:46.085231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:50.820076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":5961,"num_write_kb":189699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.956384+0000","last_change":"2024-05-04T17:35:59.221521+0000","last_active":"2024-05-04T17:38:36.956384+0000","last_peered":"2024-05-04T17:38:36.956384+0000","last_clean":"2024-05-04T17:38:36.956384+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:38:36.956384+0000","last_undegraded":"2024-05-04T17:38:36.956384+0000","last_fullsized":"2024-05-04T17:38:36.956384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:11:32.394654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'6176","reported_seq":6197,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.054893+0000","last_change":"2024-05-04T17:36:05.247978+0000","last_active":"2024-05-04T17:39:42.054893+0000","last_peered":"2024-05-04T17:39:42.054893+0000","last_clean":"2024-05-04T17:39:42.054893+0000","last_became_active":"2024-05-04T17:36:05.247697+0000","last_became_peered":"2024-05-04T17:36:05.247697+0000","last_unstale":"2024-05-04T17:39:42.054893+0000","last_undegraded":"2024-05-04T17:39:42.054893+0000","last_fullsized":"2024-05-04T17:39:42.054893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:06:15.551179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78672920,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":6162,"num_write_kb":192039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.953151+0000","last_change":"2024-05-04T17:35:59.221461+0000","last_active":"2024-05-04T17:38:36.953151+0000","last_peered":"2024-05-04T17:38:36.953151+0000","last_clean":"2024-05-04T17:38:36.953151+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:38:36.953151+0000","last_undegraded":"2024-05-04T17:38:36.953151+0000","last_fullsized":"2024-05-04T17:38:36.953151+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:44:09.888594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'3984","reported_seq":4008,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.150017+0000","last_change":"2024-05-04T17:36:05.262127+0000","last_active":"2024-05-04T17:38:37.150017+0000","last_peered":"2024-05-04T17:38:37.150017+0000","last_clean":"2024-05-04T17:38:37.150017+0000","last_became_active":"2024-05-04T17:36:05.261828+0000","last_became_peered":"2024-05-04T17:36:05.261828+0000","last_unstale":"2024-05-04T17:38:37.150017+0000","last_undegraded":"2024-05-04T17:38:37.150017+0000","last_fullsized":"2024-05-04T17:38:37.150017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:57:34.308542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":3961,"num_write_kb":185542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929862+0000","last_change":"2024-05-04T17:35:59.222186+0000","last_active":"2024-05-04T17:38:36.929862+0000","last_peered":"2024-05-04T17:38:36.929862+0000","last_clean":"2024-05-04T17:38:36.929862+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:38:36.929862+0000","last_undegraded":"2024-05-04T17:38:36.929862+0000","last_fullsized":"2024-05-04T17:38:36.929862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:15:38.191128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'5132","reported_seq":5149,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:54.216371+0000","last_change":"2024-05-04T17:36:05.395570+0000","last_active":"2024-05-04T17:38:54.216371+0000","last_peered":"2024-05-04T17:38:54.216371+0000","last_clean":"2024-05-04T17:38:54.216371+0000","last_became_active":"2024-05-04T17:36:05.395267+0000","last_became_peered":"2024-05-04T17:36:05.395267+0000","last_unstale":"2024-05-04T17:38:54.216371+0000","last_undegraded":"2024-05-04T17:38:54.216371+0000","last_fullsized":"2024-05-04T17:38:54.216371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:48:40.897417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":103041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.952916+0000","last_change":"2024-05-04T17:35:59.218744+0000","last_active":"2024-05-04T17:38:36.952916+0000","last_peered":"2024-05-04T17:38:36.952916+0000","last_clean":"2024-05-04T17:38:36.952916+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:38:36.952916+0000","last_undegraded":"2024-05-04T17:38:36.952916+0000","last_fullsized":"2024-05-04T17:38:36.952916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:37:18.872091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'6471","reported_seq":6519,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.069551+0000","last_change":"2024-05-04T17:36:05.254630+0000","last_active":"2024-05-04T17:39:42.069551+0000","last_peered":"2024-05-04T17:39:42.069551+0000","last_clean":"2024-05-04T17:39:42.069551+0000","last_became_active":"2024-05-04T17:36:05.254049+0000","last_became_peered":"2024-05-04T17:36:05.254049+0000","last_unstale":"2024-05-04T17:39:42.069551+0000","last_undegraded":"2024-05-04T17:39:42.069551+0000","last_fullsized":"2024-05-04T17:39:42.069551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:59:05.699546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123508560,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":117,"num_read_kb":157,"num_write":6516,"num_write_kb":216324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'4380","reported_seq":4402,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922479+0000","last_change":"2024-05-04T17:36:05.260042+0000","last_active":"2024-05-04T17:38:36.922479+0000","last_peered":"2024-05-04T17:38:36.922479+0000","last_clean":"2024-05-04T17:38:36.922479+0000","last_became_active":"2024-05-04T17:36:05.259684+0000","last_became_peered":"2024-05-04T17:36:05.259684+0000","last_unstale":"2024-05-04T17:38:36.922479+0000","last_undegraded":"2024-05-04T17:38:36.922479+0000","last_fullsized":"2024-05-04T17:38:36.922479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:54.517545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829632,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":4394,"num_write_kb":197895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.955303+0000","last_change":"2024-05-04T17:35:59.223094+0000","last_active":"2024-05-04T17:38:36.955303+0000","last_peered":"2024-05-04T17:38:36.955303+0000","last_clean":"2024-05-04T17:38:36.955303+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:38:36.955303+0000","last_undegraded":"2024-05-04T17:38:36.955303+0000","last_fullsized":"2024-05-04T17:38:36.955303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:54:23.823020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"22'3964","reported_seq":3981,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.149971+0000","last_change":"2024-05-04T17:36:05.395580+0000","last_active":"2024-05-04T17:38:37.149971+0000","last_peered":"2024-05-04T17:38:37.149971+0000","last_clean":"2024-05-04T17:38:37.149971+0000","last_became_active":"2024-05-04T17:36:05.395272+0000","last_became_peered":"2024-05-04T17:36:05.395272+0000","last_unstale":"2024-05-04T17:38:37.149971+0000","last_undegraded":"2024-05-04T17:38:37.149971+0000","last_fullsized":"2024-05-04T17:38:37.149971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:23:05.867612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110366368,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":158385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.926727+0000","last_change":"2024-05-04T17:35:59.331384+0000","last_active":"2024-05-04T17:38:36.926727+0000","last_peered":"2024-05-04T17:38:36.926727+0000","last_clean":"2024-05-04T17:38:36.926727+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:38:36.926727+0000","last_undegraded":"2024-05-04T17:38:36.926727+0000","last_fullsized":"2024-05-04T17:38:36.926727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:24:22.601115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"22'4100","reported_seq":4117,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.099902+0000","last_change":"2024-05-04T17:36:05.261680+0000","last_active":"2024-05-04T17:38:37.099902+0000","last_peered":"2024-05-04T17:38:37.099902+0000","last_clean":"2024-05-04T17:38:37.099902+0000","last_became_active":"2024-05-04T17:36:05.261395+0000","last_became_peered":"2024-05-04T17:36:05.261395+0000","last_unstale":"2024-05-04T17:38:37.099902+0000","last_undegraded":"2024-05-04T17:38:37.099902+0000","last_fullsized":"2024-05-04T17:38:37.099902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:02:29.609251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106299392,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":186884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929479+0000","last_change":"2024-05-04T17:35:59.223322+0000","last_active":"2024-05-04T17:38:36.929479+0000","last_peered":"2024-05-04T17:38:36.929479+0000","last_clean":"2024-05-04T17:38:36.929479+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:38:36.929479+0000","last_undegraded":"2024-05-04T17:38:36.929479+0000","last_fullsized":"2024-05-04T17:38:36.929479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:34:10.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"22'5762","reported_seq":5779,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:50.022039+0000","last_change":"2024-05-04T17:36:05.258017+0000","last_active":"2024-05-04T17:38:50.022039+0000","last_peered":"2024-05-04T17:38:50.022039+0000","last_clean":"2024-05-04T17:38:50.022039+0000","last_became_active":"2024-05-04T17:36:05.257750+0000","last_became_peered":"2024-05-04T17:36:05.257750+0000","last_unstale":"2024-05-04T17:38:50.022039+0000","last_undegraded":"2024-05-04T17:38:50.022039+0000","last_fullsized":"2024-05-04T17:38:50.022039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:20:27.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":181199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.954319+0000","last_change":"2024-05-04T17:35:59.221969+0000","last_active":"2024-05-04T17:38:36.954319+0000","last_peered":"2024-05-04T17:38:36.954319+0000","last_clean":"2024-05-04T17:38:36.954319+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:38:36.954319+0000","last_undegraded":"2024-05-04T17:38:36.954319+0000","last_fullsized":"2024-05-04T17:38:36.954319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:22:19.190308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929485+0000","last_change":"2024-05-04T17:35:58.172775+0000","last_active":"2024-05-04T17:38:36.929485+0000","last_peered":"2024-05-04T17:38:36.929485+0000","last_clean":"2024-05-04T17:38:36.929485+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:38:36.929485+0000","last_undegraded":"2024-05-04T17:38:36.929485+0000","last_fullsized":"2024-05-04T17:38:36.929485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_clean_scrub_stamp":"2024-05-04T17:35:57.156954+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:43:38.793258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"22'2730","reported_seq":2758,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148498+0000","last_change":"2024-05-04T17:36:05.250179+0000","last_active":"2024-05-04T17:38:37.148498+0000","last_peered":"2024-05-04T17:38:37.148498+0000","last_clean":"2024-05-04T17:38:37.148498+0000","last_became_active":"2024-05-04T17:36:05.249926+0000","last_became_peered":"2024-05-04T17:36:05.249926+0000","last_unstale":"2024-05-04T17:38:37.148498+0000","last_undegraded":"2024-05-04T17:38:37.148498+0000","last_fullsized":"2024-05-04T17:38:37.148498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:51:15.283094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":2682,"num_write_kb":127745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"22'9936","reported_seq":10067,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:53.646563+0000","last_change":"2024-05-04T17:36:05.395659+0000","last_active":"2024-05-04T17:38:53.646563+0000","last_peered":"2024-05-04T17:38:53.646563+0000","last_clean":"2024-05-04T17:38:53.646563+0000","last_became_active":"2024-05-04T17:36:05.395416+0000","last_became_peered":"2024-05-04T17:36:05.395416+0000","last_unstale":"2024-05-04T17:38:53.646563+0000","last_undegraded":"2024-05-04T17:38:53.646563+0000","last_fullsized":"2024-05-04T17:38:53.646563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:18:49.259604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":361,"num_read_kb":322,"num_write":6850,"num_write_kb":187020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"22'4846","reported_seq":4866,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:54.291500+0000","last_change":"2024-05-04T17:36:05.257442+0000","last_active":"2024-05-04T17:38:54.291500+0000","last_peered":"2024-05-04T17:38:54.291500+0000","last_clean":"2024-05-04T17:38:54.291500+0000","last_became_active":"2024-05-04T17:36:05.257078+0000","last_became_peered":"2024-05-04T17:36:05.257078+0000","last_unstale":"2024-05-04T17:38:54.291500+0000","last_undegraded":"2024-05-04T17:38:54.291500+0000","last_fullsized":"2024-05-04T17:38:54.291500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:53:55.417005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4826,"num_write_kb":165126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"21'4340","reported_seq":4360,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.922158+0000","last_change":"2024-05-04T17:36:05.250165+0000","last_active":"2024-05-04T17:38:36.922158+0000","last_peered":"2024-05-04T17:38:36.922158+0000","last_clean":"2024-05-04T17:38:36.922158+0000","last_became_active":"2024-05-04T17:36:05.249845+0000","last_became_peered":"2024-05-04T17:36:05.249845+0000","last_unstale":"2024-05-04T17:38:36.922158+0000","last_undegraded":"2024-05-04T17:38:36.922158+0000","last_fullsized":"2024-05-04T17:38:36.922158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:40:29.446049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":4449,"num_write_kb":193090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"22'5660","reported_seq":5802,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148711+0000","last_change":"2024-05-04T17:36:05.258056+0000","last_active":"2024-05-04T17:38:37.148711+0000","last_peered":"2024-05-04T17:38:37.148711+0000","last_clean":"2024-05-04T17:38:37.148711+0000","last_became_active":"2024-05-04T17:36:05.257760+0000","last_became_peered":"2024-05-04T17:36:05.257760+0000","last_unstale":"2024-05-04T17:38:37.148711+0000","last_undegraded":"2024-05-04T17:38:37.148711+0000","last_fullsized":"2024-05-04T17:38:37.148711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:21:44.622756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140053176,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":196,"num_read_kb":182,"num_write":5664,"num_write_kb":217362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"22'3531","reported_seq":3565,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.098371+0000","last_change":"2024-05-04T17:36:05.261546+0000","last_active":"2024-05-04T17:38:37.098371+0000","last_peered":"2024-05-04T17:38:37.098371+0000","last_clean":"2024-05-04T17:38:37.098371+0000","last_became_active":"2024-05-04T17:36:05.261259+0000","last_became_peered":"2024-05-04T17:36:05.261259+0000","last_unstale":"2024-05-04T17:38:37.098371+0000","last_undegraded":"2024-05-04T17:38:37.098371+0000","last_fullsized":"2024-05-04T17:38:37.098371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:56:45.912709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":66,"num_read_kb":115,"num_write":3541,"num_write_kb":181187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"22'4849","reported_seq":4934,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.189601+0000","last_change":"2024-05-04T17:36:05.263514+0000","last_active":"2024-05-04T17:38:37.189601+0000","last_peered":"2024-05-04T17:38:37.189601+0000","last_clean":"2024-05-04T17:38:37.189601+0000","last_became_active":"2024-05-04T17:36:05.263214+0000","last_became_peered":"2024-05-04T17:36:05.263214+0000","last_unstale":"2024-05-04T17:38:37.189601+0000","last_undegraded":"2024-05-04T17:38:37.189601+0000","last_fullsized":"2024-05-04T17:38:37.189601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:05.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121643536,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":259,"num_read_kb":194,"num_write":5097,"num_write_kb":162415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'3517","reported_seq":3534,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:39:42.056418+0000","last_change":"2024-05-04T17:36:05.263191+0000","last_active":"2024-05-04T17:39:42.056418+0000","last_peered":"2024-05-04T17:39:42.056418+0000","last_clean":"2024-05-04T17:39:42.056418+0000","last_became_active":"2024-05-04T17:36:05.262817+0000","last_became_peered":"2024-05-04T17:36:05.262817+0000","last_unstale":"2024-05-04T17:39:42.056418+0000","last_undegraded":"2024-05-04T17:39:42.056418+0000","last_fullsized":"2024-05-04T17:39:42.056418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:17:30.844586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80883712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":134410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"22'3809","reported_seq":3828,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.148181+0000","last_change":"2024-05-04T17:36:05.250207+0000","last_active":"2024-05-04T17:38:37.148181+0000","last_peered":"2024-05-04T17:38:37.148181+0000","last_clean":"2024-05-04T17:38:37.148181+0000","last_became_active":"2024-05-04T17:36:05.249932+0000","last_became_peered":"2024-05-04T17:36:05.249932+0000","last_unstale":"2024-05-04T17:38:37.148181+0000","last_undegraded":"2024-05-04T17:38:37.148181+0000","last_fullsized":"2024-05-04T17:38:37.148181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:23:37.128989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112574992,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3768,"num_write_kb":163450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"21'3909","reported_seq":3933,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:36.929503+0000","last_change":"2024-05-04T17:36:05.263401+0000","last_active":"2024-05-04T17:38:36.929503+0000","last_peered":"2024-05-04T17:38:36.929503+0000","last_clean":"2024-05-04T17:38:36.929503+0000","last_became_active":"2024-05-04T17:36:05.263000+0000","last_became_peered":"2024-05-04T17:36:05.263000+0000","last_unstale":"2024-05-04T17:38:36.929503+0000","last_undegraded":"2024-05-04T17:38:36.929503+0000","last_fullsized":"2024-05-04T17:38:36.929503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3909,"log_dups_size":0,"ondisk_log_size":3909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:51:16.339176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":3923,"num_write_kb":150485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"22'3427","reported_seq":3457,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:37.150540+0000","last_change":"2024-05-04T17:36:05.261174+0000","last_active":"2024-05-04T17:38:37.150540+0000","last_peered":"2024-05-04T17:38:37.150540+0000","last_clean":"2024-05-04T17:38:37.150540+0000","last_became_active":"2024-05-04T17:36:05.260881+0000","last_became_peered":"2024-05-04T17:36:05.260881+0000","last_unstale":"2024-05-04T17:38:37.150540+0000","last_undegraded":"2024-05-04T17:38:37.150540+0000","last_fullsized":"2024-05-04T17:38:37.150540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:28:46.613430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":54,"num_read_kb":41,"num_write":3408,"num_write_kb":117896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'8241","reported_seq":8261,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-04T17:38:48.823498+0000","last_change":"2024-05-04T17:36:05.263104+0000","last_active":"2024-05-04T17:38:48.823498+0000","last_peered":"2024-05-04T17:38:48.823498+0000","last_clean":"2024-05-04T17:38:48.823498+0000","last_became_active":"2024-05-04T17:36:05.262502+0000","last_became_peered":"2024-05-04T17:36:05.262502+0000","last_unstale":"2024-05-04T17:38:48.823498+0000","last_undegraded":"2024-05-04T17:38:48.823498+0000","last_fullsized":"2024-05-04T17:38:48.823498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:53:50.786398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8220,"num_write_kb":197892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3578544904,"num_objects":881,"num_object_clones":0,"num_object_copies":1762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":2504,"num_read_kb":2515,"num_write":163614,"num_write_kb":5652193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6859563008,"data_stored":6855288984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166464,"ondisk_log_size":166464,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542192,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343508,"kb_used_data":1055564,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93028332,"statfs":{"total":96636764160,"available":95261011968,"internally_reserved":0,"allocated":1080897536,"data_stored":1080011519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":294844874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542194,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266616,"kb_used_data":1005040,"kb_used_omap":1,"kb_used_meta":261566,"kb_avail":93105224,"statfs":{"total":96636764160,"available":95339749376,"internally_reserved":0,"allocated":1029160960,"data_stored":1028421251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267844037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542194,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1724732,"kb_used_data":1350772,"kb_used_omap":1,"kb_used_meta":373950,"kb_avail":92647108,"statfs":{"total":96636764160,"available":94870638592,"internally_reserved":0,"allocated":1383190528,"data_stored":1382090015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":382925253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401612,"kb_used_data":330244,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93970228,"statfs":{"total":96636764160,"available":96225513472,"internally_reserved":0,"allocated":338169856,"data_stored":337866195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362136,"kb_used_data":1074192,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93009704,"statfs":{"total":96636764160,"available":95241936896,"internally_reserved":0,"allocated":1099972608,"data_stored":1099151903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294844871},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377668,"kb_used_data":288188,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93994172,"statfs":{"total":96636764160,"available":96250032128,"internally_reserved":0,"allocated":295104512,"data_stored":294760883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649108,"kb_used_data":545804,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93722732,"statfs":{"total":96636764160,"available":95972077568,"internally_reserved":0,"allocated":558903296,"data_stored":558497191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1317396,"kb_used_data":1051660,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":93054444,"statfs":{"total":96636764160,"available":95287750656,"internally_reserved":0,"allocated":1076899840,"data_stored":1076106627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272103877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1076158464,"data_stored":1075507856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":558694400,"data_stored":558427348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294895616,"data_stored":294691040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1099763712,"data_stored":1099082060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1382981632,"data_stored":1382020172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1028952064,"data_stored":1028351408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":337960960,"data_stored":337796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080156160,"data_stored":1079412748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:39:54.510 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:39:54.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.605+0000 7faa0920f640 1 Processor -- start 2024-05-04T17:39:54.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa0920f640 1 -- start start 2024-05-04T17:39:54.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa0920f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa04146460 con 0x7faa04146710 2024-05-04T17:39:54.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa0920f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa04073120 con 0x7faa04142340 2024-05-04T17:39:54.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa0920f640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa04071400 con 0x7faa04146080 2024-05-04T17:39:54.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa02d9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7faa04146080 0x7faa0406e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36742/0 (socket says 172.21.15.110:36742) 2024-05-04T17:39:54.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa02d9d640 1 -- 172.21.15.110:0/2371617994 learned_addr learned my addr 172.21.15.110:0/2371617994 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.606+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307318698 0 0) 0x7faa04071400 con 0x7faa04146080 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9ec003520 con 0x7faa04146080 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2949906095 0 0) 0x7faa04146460 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa04071400 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3788857028 0 0) 0x7faa04071400 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 >> v1:172.21.15.110:6790/0 conn(0x7faa04146080 legacy=0x7faa0406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 >> v1:172.21.15.136:6789/0 conn(0x7faa04142340 legacy=0x7faa0414e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa04071f60 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa9f8001990 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9f8001b80 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa01d9b640 1 -- 172.21.15.110:0/2371617994 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa9f8001f90 con 0x7faa04146710 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa0920f640 1 -- 172.21.15.110:0/2371617994 >> v1:172.21.15.110:6789/0 conn(0x7faa04146710 legacy=0x7faa04141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa0920f640 1 -- 172.21.15.110:0/2371617994 shutdown_connections 2024-05-04T17:39:54.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa0920f640 1 -- 172.21.15.110:0/2371617994 >> 172.21.15.110:0/2371617994 conn(0x7faa040b8920 msgr2=0x7faa040b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:54.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa0920f640 1 -- 172.21.15.110:0/2371617994 shutdown_connections 2024-05-04T17:39:54.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.607+0000 7faa0920f640 1 -- 172.21.15.110:0/2371617994 wait complete. 2024-05-04T17:39:54.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0920f640 1 Processor -- start 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0920f640 1 -- start start 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0920f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa04130020 con 0x7faa04146710 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0920f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0413d5a0 con 0x7faa04142340 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0920f640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0413b010 con 0x7faa04146080 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0359e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faa04146710 0x7faa0412f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51910/0 (socket says 172.21.15.110:51910) 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7faa0359e640 1 -- 172.21.15.110:0/2710272133 learned_addr learned my addr 172.21.15.110:0/2710272133 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.609+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3833314340 0 0) 0x7faa0413b010 con 0x7faa04146080 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9f0003520 con 0x7faa04146080 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364886329 0 0) 0x7faa04130020 con 0x7faa04146710 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa0413b010 con 0x7faa04146710 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3334809146 0 0) 0x7faa0413d5a0 con 0x7faa04142340 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa04130020 con 0x7faa04142340 2024-05-04T17:39:54.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3278338545 0 0) 0x7fa9f0003520 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa0413d5a0 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa9f8001f90 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4124252807 0 0) 0x7faa0413b010 con 0x7faa04146710 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9f0003520 con 0x7faa04146710 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa9f4001880 con 0x7faa04146710 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 246889029 0 0) 0x7faa04130020 con 0x7faa04142340 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa0413b010 con 0x7faa04142340 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 853609053 0 0) 0x7faa0413d5a0 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 >> v1:172.21.15.136:6789/0 conn(0x7faa04142340 legacy=0x7faa04139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 >> v1:172.21.15.110:6789/0 conn(0x7faa04146710 legacy=0x7faa0412f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa04135d90 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa0413a1a0 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7faa0413b6f0 con 0x7faa04146080 2024-05-04T17:39:54.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9f8003e20 con 0x7faa04146080 2024-05-04T17:39:54.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.610+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fa9f8003fd0 con 0x7faa04146080 2024-05-04T17:39:54.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.611+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e4005080 con 0x7faa04146080 2024-05-04T17:39:54.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.611+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fa9f8004170 con 0x7faa04146080 2024-05-04T17:39:54.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.612+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fa9f80856a0 con 0x7faa04146080 2024-05-04T17:39:54.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.616+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fa9f8054380 con 0x7faa04146080 2024-05-04T17:39:54.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.789+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9e4005370 con 0x7faa04146080 2024-05-04T17:39:54.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.790+0000 7fa9eb7fe640 1 -- 172.21.15.110:0/2710272133 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+14700 (unknown 3451294859 0 1948710397) 0x7fa9f8053c80 con 0x7faa04146080 2024-05-04T17:39:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:39:54.792 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"5cbb7704-6b22-42de-b82d-75ed642c3bca","created":"2024-05-04T17:35:47.067563+0000","modified":"2024-05-04T17:39:53.031883+0000","last_up_change":"2024-05-04T17:35:57.156954+0000","last_in_change":"2024-05-04T17:35:49.180095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:35:57.156872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-04T17:35:57.779672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephsqlite","create_time":"2024-05-04T17:36:04.054881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac107878-2a9c-435d-a24b-634939607eb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6800","nonce":1712129713}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6801","nonce":1712129713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6803","nonce":1712129713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6802","nonce":1712129713}]},"public_addr":"172.21.15.110:6800/1712129713","cluster_addr":"172.21.15.110:6801/1712129713","heartbeat_back_addr":"172.21.15.110:6803/1712129713","heartbeat_front_addr":"172.21.15.110:6802/1712129713","state":["exists","up"]},{"osd":1,"uuid":"895a609b-83a1-4d9d-a323-9e777df46326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6808","nonce":1520708426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6810","nonce":1520708426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6814","nonce":1520708426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6812","nonce":1520708426}]},"public_addr":"172.21.15.110:6808/1520708426","cluster_addr":"172.21.15.110:6810/1520708426","heartbeat_back_addr":"172.21.15.110:6814/1520708426","heartbeat_front_addr":"172.21.15.110:6812/1520708426","state":["exists","up"]},{"osd":2,"uuid":"be5c569a-c157-48d0-b8d0-f44918cb1204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6804","nonce":4241686922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6805","nonce":4241686922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6807","nonce":4241686922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6806","nonce":4241686922}]},"public_addr":"172.21.15.110:6804/4241686922","cluster_addr":"172.21.15.110:6805/4241686922","heartbeat_back_addr":"172.21.15.110:6807/4241686922","heartbeat_front_addr":"172.21.15.110:6806/4241686922","state":["exists","up"]},{"osd":3,"uuid":"ff7aaeae-b017-4ece-8dab-2adea4056df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6809","nonce":2632237368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6811","nonce":2632237368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6815","nonce":2632237368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.110:6813","nonce":2632237368}]},"public_addr":"172.21.15.110:6809/2632237368","cluster_addr":"172.21.15.110:6811/2632237368","heartbeat_back_addr":"172.21.15.110:6815/2632237368","heartbeat_front_addr":"172.21.15.110:6813/2632237368","state":["exists","up"]},{"osd":4,"uuid":"98a44aae-f478-4906-9f02-c63ae35f54d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6804","nonce":2025414972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6805","nonce":2025414972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6807","nonce":2025414972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6806","nonce":2025414972}]},"public_addr":"172.21.15.136:6804/2025414972","cluster_addr":"172.21.15.136:6805/2025414972","heartbeat_back_addr":"172.21.15.136:6807/2025414972","heartbeat_front_addr":"172.21.15.136:6806/2025414972","state":["exists","up"]},{"osd":5,"uuid":"c625ba28-d2ca-4ecb-8de2-4fecc934e626","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6800","nonce":2872378682}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6801","nonce":2872378682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6803","nonce":2872378682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6802","nonce":2872378682}]},"public_addr":"172.21.15.136:6800/2872378682","cluster_addr":"172.21.15.136:6801/2872378682","heartbeat_back_addr":"172.21.15.136:6803/2872378682","heartbeat_front_addr":"172.21.15.136:6802/2872378682","state":["exists","up"]},{"osd":6,"uuid":"ea7e6cf8-ebd4-4fc5-9cf0-bfae1c408373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6812","nonce":239937373}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6813","nonce":239937373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6815","nonce":239937373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6814","nonce":239937373}]},"public_addr":"172.21.15.136:6812/239937373","cluster_addr":"172.21.15.136:6813/239937373","heartbeat_back_addr":"172.21.15.136:6815/239937373","heartbeat_front_addr":"172.21.15.136:6814/239937373","state":["exists","up"]},{"osd":7,"uuid":"db593666-b9e5-4cee-b022-f243b58345cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6808","nonce":2884573928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6809","nonce":2884573928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6811","nonce":2884573928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.136:6810","nonce":2884573928}]},"public_addr":"172.21.15.136:6808/2884573928","cluster_addr":"172.21.15.136:6809/2884573928","heartbeat_back_addr":"172.21.15.136:6811/2884573928","heartbeat_front_addr":"172.21.15.136:6810/2884573928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.320324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.405374+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.374617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.391393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.323092+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.319483+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.375307+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:35:54.301421+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:0/1554294182":"2024-05-04T18:39:52.085204+0000","172.21.15.110:0/2079500517":"2024-05-04T18:38:36.601856+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:39:54.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.792+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 >> v1:172.21.15.110:6816/783516658 conn(0x7fa9f006c970 legacy=0x7fa9f006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.792+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 >> v1:172.21.15.110:6790/0 conn(0x7faa04146080 legacy=0x7faa0412f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:54.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.793+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 shutdown_connections 2024-05-04T17:39:54.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.793+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 >> 172.21.15.110:0/2710272133 conn(0x7faa040b8920 msgr2=0x7faa040b8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:54.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.793+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 shutdown_connections 2024-05-04T17:39:54.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:54.793+0000 7faa0920f640 1 -- 172.21.15.110:0/2710272133 wait complete. 2024-05-04T17:39:55.806 INFO:tasks.ceph:Scrubbing osd.0 2024-05-04T17:39:55.807 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:55.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.917+0000 7fb7ecb5b640 1 Processor -- start 2024-05-04T17:39:55.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.917+0000 7fb7ecb5b640 1 -- start start 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7ecb5b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80b0050 con 0x7fb7e8143380 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7ecb5b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e8061e70 con 0x7fb7e80afc70 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7ecb5b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e806fd10 con 0x7fb7e8142840 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7e6d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb7e8143380 0x7fb7e814e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50946/0 (socket says 172.21.15.110:50946) 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7e6d9d640 1 -- 172.21.15.110:0/1018466549 learned_addr learned my addr 172.21.15.110:0/1018466549 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820255366 0 0) 0x7fb7e80b0050 con 0x7fb7e8143380 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7d0003520 con 0x7fb7e8143380 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1933200773 0 0) 0x7fb7e806fd10 con 0x7fb7e8142840 2024-05-04T17:39:55.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.918+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e80b0050 con 0x7fb7e8142840 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2574277860 0 0) 0x7fb7e8061e70 con 0x7fb7e80afc70 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e806fd10 con 0x7fb7e80afc70 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2572513759 0 0) 0x7fb7d0003520 con 0x7fb7e8143380 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 >> v1:172.21.15.110:6790/0 conn(0x7fb7e8142840 legacy=0x7fb7e80ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 >> v1:172.21.15.136:6789/0 conn(0x7fb7e80afc70 legacy=0x7fb7e813c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e806fff0 con 0x7fb7e8143380 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb7e0001990 con 0x7fb7e8143380 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7e0001c40 con 0x7fb7e8143380 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7e559a640 1 -- 172.21.15.110:0/1018466549 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb7e0002050 con 0x7fb7e8143380 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/1018466549 >> v1:172.21.15.110:6789/0 conn(0x7fb7e8143380 legacy=0x7fb7e814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/1018466549 shutdown_connections 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/1018466549 >> 172.21.15.110:0/1018466549 conn(0x7fb7e80b0e30 msgr2=0x7fb7e80b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:55.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.919+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/1018466549 shutdown_connections 2024-05-04T17:39:55.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.920+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/1018466549 wait complete. 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7ecb5b640 1 Processor -- start 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7ecb5b640 1 -- start start 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7ecb5b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e81dbd00 con 0x7fb7e8142840 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7ecb5b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e8145960 con 0x7fb7e8143380 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7ecb5b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e8136d10 con 0x7fb7e80afc70 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7e659c640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb7e80afc70 0x7fb7e8136600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40950/0 (socket says 172.21.15.110:40950) 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7e5d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb7e8142840 0x7fb7e81daee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50952/0 (socket says 172.21.15.110:50952) 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.921+0000 7fb7e659c640 1 -- 172.21.15.110:0/2890417162 learned_addr learned my addr 172.21.15.110:0/2890417162 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036957187 0 0) 0x7fb7e8136d10 con 0x7fb7e80afc70 2024-05-04T17:39:55.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7c8003520 con 0x7fb7e80afc70 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 303918026 0 0) 0x7fb7e81dbd00 con 0x7fb7e8142840 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e8136d10 con 0x7fb7e8142840 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973177754 0 0) 0x7fb7e8145960 con 0x7fb7e8143380 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e81dbd00 con 0x7fb7e8143380 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 451624044 0 0) 0x7fb7c8003520 con 0x7fb7e80afc70 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7e8145960 con 0x7fb7e80afc70 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb7d8001930 con 0x7fb7e80afc70 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1948491287 0 0) 0x7fb7e8136d10 con 0x7fb7e8142840 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7c8003520 con 0x7fb7e8142840 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb7dc001990 con 0x7fb7e8142840 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3320948738 0 0) 0x7fb7e81dbd00 con 0x7fb7e8143380 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7e8136d10 con 0x7fb7e8143380 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb7e0001df0 con 0x7fb7e8143380 2024-05-04T17:39:55.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1578988980 0 0) 0x7fb7e8145960 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 >> v1:172.21.15.136:6789/0 conn(0x7fb7e8143380 legacy=0x7fb7e81db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 >> v1:172.21.15.110:6789/0 conn(0x7fb7e8142840 legacy=0x7fb7e81daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e8136ee0 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e8072360 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7e8145ff0 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7d8001d00 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.922+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb7d8003a60 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.923+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fb7cc000f80 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.924+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fb7d8003a60 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.924+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fb7d80861a0 con 0x7fb7e80afc70 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.924+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6800/1712129713 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7c8078450 con 0x7fb7c80747e0 2024-05-04T17:39:55.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.924+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7fb7d8086670 con 0x7fb7e80afc70 2024-05-04T17:39:55.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.925+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== osd.0 v1:172.21.15.110:6800/1712129713 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb7c8078450 con 0x7fb7c80747e0 2024-05-04T17:39:55.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.938+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 --> v1:172.21.15.110:6800/1712129713 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb7cc002cb0 con 0x7fb7c80747e0 2024-05-04T17:39:55.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.938+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/2890417162 <== osd.0 v1:172.21.15.110:6800/1712129713 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fb7cc002cb0 con 0x7fb7c80747e0 2024-05-04T17:39:55.940 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:55.940 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:55.940 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.938+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 >> v1:172.21.15.110:6800/1712129713 conn(0x7fb7c80747e0 legacy=0x7fb7c8076c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 >> v1:172.21.15.110:6816/783516658 conn(0x7fb7c806c9d0 legacy=0x7fb7c806ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 >> v1:172.21.15.110:6790/0 conn(0x7fb7e80afc70 legacy=0x7fb7e8136600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7e6d9d640 1 -- 172.21.15.110:0/2890417162 reap_dead start 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 shutdown_connections 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 >> 172.21.15.110:0/2890417162 conn(0x7fb7e80b0e30 msgr2=0x7fb7e80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 shutdown_connections 2024-05-04T17:39:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:55.939+0000 7fb7ecb5b640 1 -- 172.21.15.110:0/2890417162 wait complete. 2024-05-04T17:39:55.951 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-04T17:39:56.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.048+0000 7fb910fce640 1 Processor -- start 2024-05-04T17:39:56.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.048+0000 7fb910fce640 1 -- start start 2024-05-04T17:39:56.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb910fce640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c0622c0 con 0x7fb90c13c2d0 2024-05-04T17:39:56.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb910fce640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c0733f0 con 0x7fb90c142840 2024-05-04T17:39:56.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb910fce640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c072270 con 0x7fb90c061ee0 2024-05-04T17:39:56.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb90a59c640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb90c061ee0 0x7fb90c071040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40962/0 (socket says 172.21.15.110:40962) 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb90ad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb90c13c2d0 0x7fb90c13c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50964/0 (socket says 172.21.15.110:50964) 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb90a59c640 1 -- 172.21.15.110:0/1362205092 learned_addr learned my addr 172.21.15.110:0/1362205092 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891047752 0 0) 0x7fb90c072270 con 0x7fb90c061ee0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8f8003520 con 0x7fb90c061ee0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916574943 0 0) 0x7fb90c0622c0 con 0x7fb90c13c2d0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.049+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb90c072270 con 0x7fb90c13c2d0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1588581884 0 0) 0x7fb8f8003520 con 0x7fb90c061ee0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 >> v1:172.21.15.136:6789/0 conn(0x7fb90c142840 legacy=0x7fb90c1463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 >> v1:172.21.15.110:6789/0 conn(0x7fb90c13c2d0 legacy=0x7fb90c13c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c146ae0 con 0x7fb90c061ee0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb8fc001610 con 0x7fb90c061ee0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8fc0018f0 con 0x7fb90c061ee0 2024-05-04T17:39:56.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb909d9b640 1 -- 172.21.15.110:0/1362205092 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb8fc001d00 con 0x7fb90c061ee0 2024-05-04T17:39:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb910fce640 1 -- 172.21.15.110:0/1362205092 >> v1:172.21.15.110:6790/0 conn(0x7fb90c061ee0 legacy=0x7fb90c071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb910fce640 1 -- 172.21.15.110:0/1362205092 shutdown_connections 2024-05-04T17:39:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb910fce640 1 -- 172.21.15.110:0/1362205092 >> 172.21.15.110:0/1362205092 conn(0x7fb90c0b5420 msgr2=0x7fb90c0b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb910fce640 1 -- 172.21.15.110:0/1362205092 shutdown_connections 2024-05-04T17:39:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.050+0000 7fb910fce640 1 -- 172.21.15.110:0/1362205092 wait complete. 2024-05-04T17:39:56.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.051+0000 7fb910fce640 1 Processor -- start 2024-05-04T17:39:56.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.051+0000 7fb910fce640 1 -- start start 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb910fce640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c0b1a00 con 0x7fb90c142840 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb910fce640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c0add70 con 0x7fb90c061ee0 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb910fce640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c13f2e0 con 0x7fb90c13c2d0 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb90ad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb90c142840 0x7fb90c1e21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50976/0 (socket says 172.21.15.110:50976) 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb903fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb90c13c2d0 0x7fb90c140d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40964/0 (socket says 172.21.15.110:40964) 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb90ad9d640 1 -- 172.21.15.110:0/2511608642 learned_addr learned my addr 172.21.15.110:0/2511608642 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531462517 0 0) 0x7fb90c0b1a00 con 0x7fb90c142840 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ec003520 con 0x7fb90c142840 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1918012785 0 0) 0x7fb90c13f2e0 con 0x7fb90c13c2d0 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb90c0b1a00 con 0x7fb90c13c2d0 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614557618 0 0) 0x7fb90c0add70 con 0x7fb90c061ee0 2024-05-04T17:39:56.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.052+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb90c13f2e0 con 0x7fb90c061ee0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2706339073 0 0) 0x7fb90c0b1a00 con 0x7fb90c13c2d0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb90c0add70 con 0x7fb90c13c2d0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb8f4001810 con 0x7fb90c13c2d0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2752333862 0 0) 0x7fb8ec003520 con 0x7fb90c142840 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb90c0b1a00 con 0x7fb90c142840 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb904001500 con 0x7fb90c142840 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3682196343 0 0) 0x7fb90c13f2e0 con 0x7fb90c061ee0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8ec003520 con 0x7fb90c061ee0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb8fc001aa0 con 0x7fb90c061ee0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3039554059 0 0) 0x7fb90c0add70 con 0x7fb90c13c2d0 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 >> v1:172.21.15.136:6789/0 conn(0x7fb90c061ee0 legacy=0x7fb90c13ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 >> v1:172.21.15.110:6789/0 conn(0x7fb90c142840 legacy=0x7fb90c1e21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c13dec0 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb90c132100 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb90c13bc00 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8f4003610 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.053+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb8f40039e0 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.054+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8f0005080 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.054+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fb8f4003c00 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.055+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fb8f4086160 con 0x7fb90c13c2d0 2024-05-04T17:39:56.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.059+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fb8f4054f80 con 0x7fb90c13c2d0 2024-05-04T17:39:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.234+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fb8f0000b90 con 0x7fb8ec06ca30 2024-05-04T17:39:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.234+0000 7fb902ffd640 1 -- 172.21.15.110:0/2511608642 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7fb8f0000b90 con 0x7fb8ec06ca30 2024-05-04T17:39:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 0 to deep-scrub 2024-05-04T17:39:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.236+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 >> v1:172.21.15.110:6816/783516658 conn(0x7fb8ec06ca30 legacy=0x7fb8ec06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.236+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 >> v1:172.21.15.110:6790/0 conn(0x7fb90c13c2d0 legacy=0x7fb90c140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.236+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 shutdown_connections 2024-05-04T17:39:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.236+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 >> 172.21.15.110:0/2511608642 conn(0x7fb90c0b5420 msgr2=0x7fb90c06bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.236+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 shutdown_connections 2024-05-04T17:39:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.236+0000 7fb910fce640 1 -- 172.21.15.110:0/2511608642 wait complete. 2024-05-04T17:39:56.248 INFO:tasks.ceph:Scrubbing osd.1 2024-05-04T17:39:56.248 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.347+0000 7f8835100640 1 Processor -- start 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.347+0000 7f8835100640 1 -- start start 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f8835100640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88300a9b30 con 0x7f88300a9750 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f8835100640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88300ab460 con 0x7f88300aa1e0 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f8835100640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88300b1e20 con 0x7f883013d780 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882ed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f88300a9750 0x7f883006f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50986/0 (socket says 172.21.15.110:50986) 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882f59e640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f883013d780 0x7f883014bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40980/0 (socket says 172.21.15.110:40980) 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882ed9d640 1 -- 172.21.15.110:0/522307055 learned_addr learned my addr 172.21.15.110:0/522307055 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 59435512 0 0) 0x7f88300b1e20 con 0x7f883013d780 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f881c003520 con 0x7f883013d780 2024-05-04T17:39:56.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 334933425 0 0) 0x7f88300a9b30 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88300b1e20 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3507583298 0 0) 0x7f88300b1e20 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 >> v1:172.21.15.110:6790/0 conn(0x7f883013d780 legacy=0x7f883014bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 >> v1:172.21.15.136:6789/0 conn(0x7f88300aa1e0 legacy=0x7f8830140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8830140630 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8820001590 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88200017f0 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.348+0000 7f882e59c640 1 -- 172.21.15.110:0/522307055 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8820001bc0 con 0x7f88300a9750 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.349+0000 7f8835100640 1 -- 172.21.15.110:0/522307055 >> v1:172.21.15.110:6789/0 conn(0x7f88300a9750 legacy=0x7f883006f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.349+0000 7f8835100640 1 -- 172.21.15.110:0/522307055 shutdown_connections 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.349+0000 7f8835100640 1 -- 172.21.15.110:0/522307055 >> 172.21.15.110:0/522307055 conn(0x7f88300b0e30 msgr2=0x7f88300b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.349+0000 7f8835100640 1 -- 172.21.15.110:0/522307055 shutdown_connections 2024-05-04T17:39:56.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.349+0000 7f8835100640 1 -- 172.21.15.110:0/522307055 wait complete. 2024-05-04T17:39:56.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.350+0000 7f8835100640 1 Processor -- start 2024-05-04T17:39:56.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.350+0000 7f8835100640 1 -- start start 2024-05-04T17:39:56.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f8835100640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88300aeb00 con 0x7f88300a9750 2024-05-04T17:39:56.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f8835100640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8830138160 con 0x7f88300aa1e0 2024-05-04T17:39:56.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f8835100640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88300af270 con 0x7f883013d780 2024-05-04T17:39:56.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f882ed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f88300a9750 0x7f8830137480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50992/0 (socket says 172.21.15.110:50992) 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f882ed9d640 1 -- 172.21.15.110:0/3865443370 learned_addr learned my addr 172.21.15.110:0/3865443370 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108773444 0 0) 0x7f88300aeb00 con 0x7f88300a9750 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8810003520 con 0x7f88300a9750 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2402813810 0 0) 0x7f88300af270 con 0x7f883013d780 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88300aeb00 con 0x7f883013d780 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 857367465 0 0) 0x7f8830138160 con 0x7f88300aa1e0 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88300af270 con 0x7f88300aa1e0 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1171672304 0 0) 0x7f8810003520 con 0x7f88300a9750 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8830138160 con 0x7f88300a9750 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8820001b40 con 0x7f88300a9750 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1030887770 0 0) 0x7f88300aeb00 con 0x7f883013d780 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8810003520 con 0x7f883013d780 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8828001ad0 con 0x7f883013d780 2024-05-04T17:39:56.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 179332300 0 0) 0x7f88300af270 con 0x7f88300aa1e0 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88300aeb00 con 0x7f88300aa1e0 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8818001830 con 0x7f88300aa1e0 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4087663899 0 0) 0x7f8830138160 con 0x7f88300a9750 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 >> v1:172.21.15.110:6790/0 conn(0x7f883013d780 legacy=0x7f88301d5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 >> v1:172.21.15.136:6789/0 conn(0x7f88300aa1e0 legacy=0x7f88300acf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8830137b90 con 0x7f88300a9750 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8830137e80 con 0x7f88300a9750 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.351+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f88300ad860 con 0x7f88300a9750 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.352+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88200017f0 con 0x7f88300a9750 2024-05-04T17:39:56.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.352+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f8820004d90 con 0x7f88300a9750 2024-05-04T17:39:56.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.352+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8814000f80 con 0x7f88300a9750 2024-05-04T17:39:56.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.353+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f8820004d90 con 0x7f88300a9750 2024-05-04T17:39:56.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.353+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f88200861e0 con 0x7f88300a9750 2024-05-04T17:39:56.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.353+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6808/1520708426 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8810078450 con 0x7f8810074880 2024-05-04T17:39:56.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.353+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7f88200866b0 con 0x7f88300a9750 2024-05-04T17:39:56.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.354+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== osd.1 v1:172.21.15.110:6808/1520708426 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8810078450 con 0x7f8810074880 2024-05-04T17:39:56.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.367+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 --> v1:172.21.15.110:6808/1520708426 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8814002cb0 con 0x7f8810074880 2024-05-04T17:39:56.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.367+0000 7f88277fe640 1 -- 172.21.15.110:0/3865443370 <== osd.1 v1:172.21.15.110:6808/1520708426 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f8814002cb0 con 0x7f8810074880 2024-05-04T17:39:56.369 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:56.369 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:56.369 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 >> v1:172.21.15.110:6808/1520708426 conn(0x7f8810074880 legacy=0x7f8810076ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 >> v1:172.21.15.110:6816/783516658 conn(0x7f881006ca70 legacy=0x7f881006ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 >> v1:172.21.15.110:6789/0 conn(0x7f88300a9750 legacy=0x7f8830137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f882f59e640 1 -- 172.21.15.110:0/3865443370 reap_dead start 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 shutdown_connections 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 >> 172.21.15.110:0/3865443370 conn(0x7f88300b0e30 msgr2=0x7f883013ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 shutdown_connections 2024-05-04T17:39:56.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.368+0000 7f8835100640 1 -- 172.21.15.110:0/3865443370 wait complete. 2024-05-04T17:39:56.380 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-04T17:39:56.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.476+0000 7f295372b640 1 Processor -- start 2024-05-04T17:39:56.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.476+0000 7f295372b640 1 -- start start 2024-05-04T17:39:56.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.476+0000 7f295372b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c142860 con 0x7f294c142f50 2024-05-04T17:39:56.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f295372b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c0715c0 con 0x7f294c06f7c0 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f295372b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c13e900 con 0x7f294c142480 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2950cc6640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f294c142f50 0x7f294c06f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50998/0 (socket says 172.21.15.110:50998) 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f29514c7640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f294c142480 0x7f294c13e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40986/0 (socket says 172.21.15.110:40986) 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2950cc6640 1 -- 172.21.15.110:0/2604654279 learned_addr learned my addr 172.21.15.110:0/2604654279 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977848736 0 0) 0x7f294c13e900 con 0x7f294c142480 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2938003520 con 0x7f294c142480 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608510298 0 0) 0x7f294c142860 con 0x7f294c142f50 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c13e900 con 0x7f294c142f50 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2774374976 0 0) 0x7f294c13e900 con 0x7f294c142f50 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 >> v1:172.21.15.110:6790/0 conn(0x7f294c142480 legacy=0x7f294c13e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 >> v1:172.21.15.136:6789/0 conn(0x7f294c06f7c0 legacy=0x7f294c14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c13fca0 con 0x7f294c142f50 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f293c001610 con 0x7f294c142f50 2024-05-04T17:39:56.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f293c001870 con 0x7f294c142f50 2024-05-04T17:39:56.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.477+0000 7f2943fff640 1 -- 172.21.15.110:0/2604654279 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f293c001c80 con 0x7f294c142f50 2024-05-04T17:39:56.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.478+0000 7f295372b640 1 -- 172.21.15.110:0/2604654279 >> v1:172.21.15.110:6789/0 conn(0x7f294c142f50 legacy=0x7f294c06f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.478+0000 7f295372b640 1 -- 172.21.15.110:0/2604654279 shutdown_connections 2024-05-04T17:39:56.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.478+0000 7f295372b640 1 -- 172.21.15.110:0/2604654279 >> 172.21.15.110:0/2604654279 conn(0x7f294c0b43a0 msgr2=0x7f294c0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.478+0000 7f295372b640 1 -- 172.21.15.110:0/2604654279 shutdown_connections 2024-05-04T17:39:56.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.478+0000 7f295372b640 1 -- 172.21.15.110:0/2604654279 wait complete. 2024-05-04T17:39:56.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f295372b640 1 Processor -- start 2024-05-04T17:39:56.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f295372b640 1 -- start start 2024-05-04T17:39:56.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f295372b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c11f8b0 con 0x7f294c11dc40 2024-05-04T17:39:56.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f295372b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c11d320 con 0x7f294c142480 2024-05-04T17:39:56.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f295372b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c13cbd0 con 0x7f294c06f7c0 2024-05-04T17:39:56.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f29514c7640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f294c06f7c0 0x7f294c13c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40992/0 (socket says 172.21.15.110:40992) 2024-05-04T17:39:56.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f29514c7640 1 -- 172.21.15.110:0/2960968545 learned_addr learned my addr 172.21.15.110:0/2960968545 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.479+0000 7f2951cc8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f294c11dc40 0x7f294c1db210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51008/0 (socket says 172.21.15.110:51008) 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069610026 0 0) 0x7f294c11f8b0 con 0x7f294c11dc40 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2930003520 con 0x7f294c11dc40 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793551396 0 0) 0x7f294c13cbd0 con 0x7f294c06f7c0 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c11f8b0 con 0x7f294c06f7c0 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496687914 0 0) 0x7f294c11d320 con 0x7f294c142480 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c13cbd0 con 0x7f294c142480 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3469187480 0 0) 0x7f294c11f8b0 con 0x7f294c06f7c0 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f294c11d320 con 0x7f294c06f7c0 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2505799835 0 0) 0x7f2930003520 con 0x7f294c11dc40 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f294c11f8b0 con 0x7f294c11dc40 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f29480012c0 con 0x7f294c06f7c0 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f2944001b50 con 0x7f294c11dc40 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4176657971 0 0) 0x7f294c13cbd0 con 0x7f294c142480 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2930003520 con 0x7f294c142480 2024-05-04T17:39:56.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f293c001bc0 con 0x7f294c142480 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1469658098 0 0) 0x7f294c11d320 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 >> v1:172.21.15.136:6789/0 conn(0x7f294c142480 legacy=0x7f294c1dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 >> v1:172.21.15.110:6789/0 conn(0x7f294c11dc40 legacy=0x7f294c1db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c13cda0 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f294c11e2c0 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29480047e0 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f2948004b30 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.480+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f294c0af680 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.481+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c062e90 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.482+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f294801aec0 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.482+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f2948086240 con 0x7f294c06f7c0 2024-05-04T17:39:56.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.486+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f2948054f80 con 0x7f294c06f7c0 2024-05-04T17:39:56.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.660+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f294c13cf70 con 0x7f293006cae0 2024-05-04T17:39:56.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.660+0000 7f2941ffb640 1 -- 172.21.15.110:0/2960968545 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f294c13cf70 con 0x7f293006cae0 2024-05-04T17:39:56.662 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 1 to deep-scrub 2024-05-04T17:39:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.662+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 >> v1:172.21.15.110:6816/783516658 conn(0x7f293006cae0 legacy=0x7f293006efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.662+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 >> v1:172.21.15.110:6790/0 conn(0x7f294c06f7c0 legacy=0x7f294c13c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.662+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 shutdown_connections 2024-05-04T17:39:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.662+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 >> 172.21.15.110:0/2960968545 conn(0x7f294c0b43a0 msgr2=0x7f294c071870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.663+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 shutdown_connections 2024-05-04T17:39:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.663+0000 7f295372b640 1 -- 172.21.15.110:0/2960968545 wait complete. 2024-05-04T17:39:56.675 INFO:tasks.ceph:Scrubbing osd.2 2024-05-04T17:39:56.675 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:56.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.775+0000 7fbca82d2640 1 Processor -- start 2024-05-04T17:39:56.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.775+0000 7fbca82d2640 1 -- start start 2024-05-04T17:39:56.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.775+0000 7fbca82d2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca0073560 con 0x7fbca0071570 2024-05-04T17:39:56.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.775+0000 7fbca82d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca0072870 con 0x7fbca0148d50 2024-05-04T17:39:56.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.775+0000 7fbca82d2640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca013f440 con 0x7fbca006f980 2024-05-04T17:39:56.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca586d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbca0071570 0x7fbca00719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51014/0 (socket says 172.21.15.110:51014) 2024-05-04T17:39:56.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca606e640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fbca006f980 0x7fbca013e920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40996/0 (socket says 172.21.15.110:40996) 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca586d640 1 -- 172.21.15.110:0/2522791667 learned_addr learned my addr 172.21.15.110:0/2522791667 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831321416 0 0) 0x7fbca0073560 con 0x7fbca0071570 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc8c003520 con 0x7fbca0071570 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666421007 0 0) 0x7fbca013f440 con 0x7fbca006f980 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbca0073560 con 0x7fbca006f980 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca686f640 1 -- 172.21.15.110:0/2522791667 >> v1:172.21.15.136:6789/0 conn(0x7fbca0148d50 legacy=0x7fbca014b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca686f640 1 -- 172.21.15.110:0/2522791667 >> v1:172.21.15.136:6789/0 conn(0x7fbca0148d50 legacy=0x7fbca014b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca686f640 1 --1- 172.21.15.110:0/2522791667 >> v1:172.21.15.136:6789/0 conn(0x7fbca0148d50 0x7fbca014b140 :-1 s=OPENED pgs=294 cs=1 l=1).handle_message read tag failed 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca686f640 1 --1- 172.21.15.110:0/2522791667 >> v1:172.21.15.136:6789/0 conn(0x7fbca0148d50 0x7fbca014b140 :-1 s=OPENED pgs=294 cs=1 l=1).fault on lossy channel, failing 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3202249708 0 0) 0x7fbca0073560 con 0x7fbca006f980 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 >> v1:172.21.15.136:6789/0 conn(0x7fbca0148d50 legacy=0x7fbca014b140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 >> v1:172.21.15.110:6789/0 conn(0x7fbca0071570 legacy=0x7fbca00719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca00730b0 con 0x7fbca006f980 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbc9c001610 con 0x7fbca006f980 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc9c0018f0 con 0x7fbca006f980 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.776+0000 7fbca506c640 1 -- 172.21.15.110:0/2522791667 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbc9c001d00 con 0x7fbca006f980 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.777+0000 7fbca82d2640 1 -- 172.21.15.110:0/2522791667 >> v1:172.21.15.110:6790/0 conn(0x7fbca006f980 legacy=0x7fbca013e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.777+0000 7fbca82d2640 1 -- 172.21.15.110:0/2522791667 shutdown_connections 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.777+0000 7fbca82d2640 1 -- 172.21.15.110:0/2522791667 >> 172.21.15.110:0/2522791667 conn(0x7fbca00b0e30 msgr2=0x7fbca00b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.777+0000 7fbca82d2640 1 -- 172.21.15.110:0/2522791667 shutdown_connections 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.777+0000 7fbca82d2640 1 -- 172.21.15.110:0/2522791667 wait complete. 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca82d2640 1 Processor -- start 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca82d2640 1 -- start start 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca82d2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca01dbbb0 con 0x7fbca006f980 2024-05-04T17:39:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca82d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca0132e60 con 0x7fbca0071570 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca82d2640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca0140da0 con 0x7fbca0148d50 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca606e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbca006f980 0x7fbca0130660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51026/0 (socket says 172.21.15.110:51026) 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.778+0000 7fbca606e640 1 -- 172.21.15.110:0/2330541006 learned_addr learned my addr 172.21.15.110:0/2330541006 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330676299 0 0) 0x7fbca0140da0 con 0x7fbca0148d50 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc84003520 con 0x7fbca0148d50 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596681340 0 0) 0x7fbca01dbbb0 con 0x7fbca006f980 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbca0140da0 con 0x7fbca006f980 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3992804460 0 0) 0x7fbc84003520 con 0x7fbca0148d50 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbca01dbbb0 con 0x7fbca0148d50 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbc98001b50 con 0x7fbca0148d50 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 781507959 0 0) 0x7fbca0132e60 con 0x7fbca0071570 2024-05-04T17:39:56.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc84003520 con 0x7fbca0071570 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2043064993 0 0) 0x7fbca0140da0 con 0x7fbca006f980 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbca0132e60 con 0x7fbca006f980 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbc9c001c40 con 0x7fbca006f980 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 832706739 0 0) 0x7fbca01dbbb0 con 0x7fbca0148d50 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 >> v1:172.21.15.136:6789/0 conn(0x7fbca0071570 legacy=0x7fbca01dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 >> v1:172.21.15.110:6789/0 conn(0x7fbca006f980 legacy=0x7fbca0130660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca0132400 con 0x7fbca0148d50 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbca0130d70 con 0x7fbca0148d50 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbca0148310 con 0x7fbca0148d50 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc98003970 con 0x7fbca0148d50 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.779+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fbc98003d40 con 0x7fbca0148d50 2024-05-04T17:39:56.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.780+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbc88000f80 con 0x7fbca0148d50 2024-05-04T17:39:56.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.780+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fbc98003ee0 con 0x7fbca0148d50 2024-05-04T17:39:56.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.781+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fbc98086550 con 0x7fbca0148d50 2024-05-04T17:39:56.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.781+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6804/4241686922 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc98004730 con 0x7fbc84074880 2024-05-04T17:39:56.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.781+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7fbc98086960 con 0x7fbca0148d50 2024-05-04T17:39:56.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.782+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== osd.2 v1:172.21.15.110:6804/4241686922 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc98004730 con 0x7fbc84074880 2024-05-04T17:39:56.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.796+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 --> v1:172.21.15.110:6804/4241686922 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbc88002cb0 con 0x7fbc84074880 2024-05-04T17:39:56.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.796+0000 7fbc96ffd640 1 -- 172.21.15.110:0/2330541006 <== osd.2 v1:172.21.15.110:6804/4241686922 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fbc88002cb0 con 0x7fbc84074880 2024-05-04T17:39:56.798 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:56.798 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:56.798 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:56.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 >> v1:172.21.15.110:6804/4241686922 conn(0x7fbc84074880 legacy=0x7fbc84076ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 >> v1:172.21.15.110:6816/783516658 conn(0x7fbc8406c9e0 legacy=0x7fbc8406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 >> v1:172.21.15.110:6790/0 conn(0x7fbca0148d50 legacy=0x7fbca01db4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca686f640 1 -- 172.21.15.110:0/2330541006 reap_dead start 2024-05-04T17:39:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 shutdown_connections 2024-05-04T17:39:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 >> 172.21.15.110:0/2330541006 conn(0x7fbca00b0e30 msgr2=0x7fbca00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 shutdown_connections 2024-05-04T17:39:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.797+0000 7fbca82d2640 1 -- 172.21.15.110:0/2330541006 wait complete. 2024-05-04T17:39:56.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-04T17:39:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.926+0000 7f6817206640 1 Processor -- start 2024-05-04T17:39:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.926+0000 7f6817206640 1 -- start start 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.926+0000 7f6817206640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6810146460 con 0x7f6810146710 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.926+0000 7f6817206640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6810073120 con 0x7f6810142340 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.926+0000 7f6817206640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6810071400 con 0x7f6810146080 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f6814fa2640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f6810146080 0x7f681006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41014/0 (socket says 172.21.15.110:41014) 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f6814fa2640 1 -- 172.21.15.110:0/3392318323 learned_addr learned my addr 172.21.15.110:0/3392318323 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680ffff640 1 --1- 172.21.15.110:0/3392318323 >> v1:172.21.15.110:6789/0 conn(0x7f6810146710 0x7f6810141c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51038/0 (socket says 172.21.15.110:51038) 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736883572 0 0) 0x7f6810146460 con 0x7f6810146710 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6800003520 con 0x7f6810146710 2024-05-04T17:39:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2924019680 0 0) 0x7f6810071400 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6810146460 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543803235 0 0) 0x7f6810073120 con 0x7f6810142340 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6810071400 con 0x7f6810142340 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2763234476 0 0) 0x7f6810146460 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 >> v1:172.21.15.136:6789/0 conn(0x7f6810142340 legacy=0x7f681014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 >> v1:172.21.15.110:6789/0 conn(0x7f6810146710 legacy=0x7f6810141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6810071f60 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6804001610 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6804001870 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.927+0000 7f680f7fe640 1 -- 172.21.15.110:0/3392318323 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6804001c00 con 0x7f6810146080 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.928+0000 7f6817206640 1 -- 172.21.15.110:0/3392318323 >> v1:172.21.15.110:6790/0 conn(0x7f6810146080 legacy=0x7f681006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.928+0000 7f6817206640 1 -- 172.21.15.110:0/3392318323 shutdown_connections 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.928+0000 7f6817206640 1 -- 172.21.15.110:0/3392318323 >> 172.21.15.110:0/3392318323 conn(0x7f68100b8920 msgr2=0x7f68100b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.928+0000 7f6817206640 1 -- 172.21.15.110:0/3392318323 shutdown_connections 2024-05-04T17:39:56.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.928+0000 7f6817206640 1 -- 172.21.15.110:0/3392318323 wait complete. 2024-05-04T17:39:56.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f6817206640 1 Processor -- start 2024-05-04T17:39:56.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f6817206640 1 -- start start 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f6817206640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681013b140 con 0x7f6810146710 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f6817206640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68100afce0 con 0x7f6810142340 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f6817206640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68101db2e0 con 0x7f6810146080 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f680ffff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f6810146080 0x7f681013a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41020/0 (socket says 172.21.15.110:41020) 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f68157a3640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6810146710 0x7f681013aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51044/0 (socket says 172.21.15.110:51044) 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.929+0000 7f680ffff640 1 -- 172.21.15.110:0/3236157228 learned_addr learned my addr 172.21.15.110:0/3236157228 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 227476849 0 0) 0x7f681013b140 con 0x7f6810146710 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f4003520 con 0x7f6810146710 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679790337 0 0) 0x7f68101db2e0 con 0x7f6810146080 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f681013b140 con 0x7f6810146080 2024-05-04T17:39:56.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3598285810 0 0) 0x7f67f4003520 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68101db2e0 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6808001c50 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2680761889 0 0) 0x7f68100afce0 con 0x7f6810142340 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f4003520 con 0x7f6810142340 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1304769927 0 0) 0x7f681013b140 con 0x7f6810146080 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68100afce0 con 0x7f6810146080 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f67fc0017a0 con 0x7f6810146080 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4273705977 0 0) 0x7f68101db2e0 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 >> v1:172.21.15.110:6790/0 conn(0x7f6810146080 legacy=0x7f681013a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 >> v1:172.21.15.136:6789/0 conn(0x7f6810142340 legacy=0x7f68101dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68100af900 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f681013b350 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f68100b01c0 con 0x7f6810146710 2024-05-04T17:39:56.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6808003970 con 0x7f6810146710 2024-05-04T17:39:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.930+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f6808003d40 con 0x7f6810146710 2024-05-04T17:39:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.931+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f0005080 con 0x7f6810146710 2024-05-04T17:39:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.932+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f6808003f60 con 0x7f6810146710 2024-05-04T17:39:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.932+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f6808086750 con 0x7f6810146710 2024-05-04T17:39:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:56.936+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f6808055490 con 0x7f6810146710 2024-05-04T17:39:57.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.115+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f67f0000b90 con 0x7f67f406c9e0 2024-05-04T17:39:57.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.115+0000 7f680d7fa640 1 -- 172.21.15.110:0/3236157228 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f67f0000b90 con 0x7f67f406c9e0 2024-05-04T17:39:57.117 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 2 to deep-scrub 2024-05-04T17:39:57.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.117+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 >> v1:172.21.15.110:6816/783516658 conn(0x7f67f406c9e0 legacy=0x7f67f406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.117+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 >> v1:172.21.15.110:6789/0 conn(0x7f6810146710 legacy=0x7f681013aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.117+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 shutdown_connections 2024-05-04T17:39:57.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.117+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 >> 172.21.15.110:0/3236157228 conn(0x7f68100b8920 msgr2=0x7f68100b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.117+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 shutdown_connections 2024-05-04T17:39:57.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.117+0000 7f6817206640 1 -- 172.21.15.110:0/3236157228 wait complete. 2024-05-04T17:39:57.130 INFO:tasks.ceph:Scrubbing osd.3 2024-05-04T17:39:57.131 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.231+0000 7f378eae0640 1 Processor -- start 2024-05-04T17:39:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.231+0000 7f378eae0640 1 -- start start 2024-05-04T17:39:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f378eae0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3788073560 con 0x7f3788071570 2024-05-04T17:39:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f378eae0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3788072870 con 0x7f378806f980 2024-05-04T17:39:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f378eae0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378813f440 con 0x7f3788148d50 2024-05-04T17:39:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f3787fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3788071570 0x7f37880719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51054/0 (socket says 172.21.15.110:51054) 2024-05-04T17:39:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f378d07d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f3788148d50 0x7f378814b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41022/0 (socket says 172.21.15.110:41022) 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f3787fff640 1 -- 172.21.15.110:0/901520819 learned_addr learned my addr 172.21.15.110:0/901520819 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133076520 0 0) 0x7f378813f440 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3770005730 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976404287 0 0) 0x7f3788073560 con 0x7f3788071570 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f378813f440 con 0x7f3788071570 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 722939778 0 0) 0x7f3770005730 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 >> v1:172.21.15.136:6789/0 conn(0x7f378806f980 legacy=0x7f378813e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 >> v1:172.21.15.110:6789/0 conn(0x7f3788071570 legacy=0x7f37880719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880730b0 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.232+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3780001990 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3780001c70 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f37877fe640 1 -- 172.21.15.110:0/901520819 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3780002080 con 0x7f3788148d50 2024-05-04T17:39:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f378eae0640 1 -- 172.21.15.110:0/901520819 >> v1:172.21.15.110:6790/0 conn(0x7f3788148d50 legacy=0x7f378814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f378eae0640 1 -- 172.21.15.110:0/901520819 shutdown_connections 2024-05-04T17:39:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f378eae0640 1 -- 172.21.15.110:0/901520819 >> 172.21.15.110:0/901520819 conn(0x7f37880b0e30 msgr2=0x7f37880b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f378eae0640 1 -- 172.21.15.110:0/901520819 shutdown_connections 2024-05-04T17:39:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.233+0000 7f378eae0640 1 -- 172.21.15.110:0/901520819 wait complete. 2024-05-04T17:39:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.234+0000 7f378eae0640 1 Processor -- start 2024-05-04T17:39:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.234+0000 7f378eae0640 1 -- start start 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f378eae0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37881d4540 con 0x7f3788148d50 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f378eae0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378812f550 con 0x7f378806f980 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f378eae0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378812f830 con 0x7f3788071570 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f3787fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f3788071570 0x7f37881d3720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41030/0 (socket says 172.21.15.110:41030) 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f3787fff640 1 -- 172.21.15.110:0/1372691452 learned_addr learned my addr 172.21.15.110:0/1372691452 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981209742 0 0) 0x7f37881d4540 con 0x7f3788148d50 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3774003520 con 0x7f3788148d50 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259244463 0 0) 0x7f378812f830 con 0x7f3788071570 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37881d4540 con 0x7f3788071570 2024-05-04T17:39:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750331115 0 0) 0x7f378812f550 con 0x7f378806f980 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f378812f830 con 0x7f378806f980 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 180383033 0 0) 0x7f3774003520 con 0x7f3788148d50 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f378812f550 con 0x7f3788148d50 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3780001fc0 con 0x7f3788148d50 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4027740618 0 0) 0x7f37881d4540 con 0x7f3788071570 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3774003520 con 0x7f3788071570 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f377c001320 con 0x7f3788071570 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2240651904 0 0) 0x7f378812f550 con 0x7f3788148d50 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 >> v1:172.21.15.110:6790/0 conn(0x7f3788071570 legacy=0x7f37881d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.235+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 >> v1:172.21.15.136:6789/0 conn(0x7f378806f980 legacy=0x7f378812d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.236+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378812e100 con 0x7f3788148d50 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.236+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f378812ffa0 con 0x7f3788148d50 2024-05-04T17:39:57.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.236+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f37881df0e0 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.236+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3780001c70 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.236+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f3780004ee0 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.236+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3768000f80 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.237+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f378001b050 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.237+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f3780086340 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.238+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6809/2632237368 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37740781b0 con 0x7f3774074540 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.238+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7f3780086810 con 0x7f3788148d50 2024-05-04T17:39:57.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.238+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== osd.3 v1:172.21.15.110:6809/2632237368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f37740781b0 con 0x7f3774074540 2024-05-04T17:39:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.251+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 --> v1:172.21.15.110:6809/2632237368 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3768002cb0 con 0x7f3774074540 2024-05-04T17:39:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.251+0000 7f37857fa640 1 -- 172.21.15.110:0/1372691452 <== osd.3 v1:172.21.15.110:6809/2632237368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f3768002cb0 con 0x7f3774074540 2024-05-04T17:39:57.253 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:57.253 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:57.253 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 >> v1:172.21.15.110:6809/2632237368 conn(0x7f3774074540 legacy=0x7f3774076960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 >> v1:172.21.15.110:6816/783516658 conn(0x7f377406c730 legacy=0x7f377406ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 >> v1:172.21.15.110:6789/0 conn(0x7f3788148d50 legacy=0x7f37881d3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378d07d640 1 -- 172.21.15.110:0/1372691452 reap_dead start 2024-05-04T17:39:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 shutdown_connections 2024-05-04T17:39:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 >> 172.21.15.110:0/1372691452 conn(0x7f37880b0e30 msgr2=0x7f37880b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 shutdown_connections 2024-05-04T17:39:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.252+0000 7f378eae0640 1 -- 172.21.15.110:0/1372691452 wait complete. 2024-05-04T17:39:57.264 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-04T17:39:57.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.361+0000 7fb29298a640 1 Processor -- start 2024-05-04T17:39:57.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.361+0000 7fb29298a640 1 -- start start 2024-05-04T17:39:57.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb29298a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c142860 con 0x7fb28c06f7c0 2024-05-04T17:39:57.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb29298a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c0715c0 con 0x7fb28c142480 2024-05-04T17:39:57.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb29298a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c13e900 con 0x7fb28c142f50 2024-05-04T17:39:57.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28b7fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb28c142f50 0x7fb28c06f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41032/0 (socket says 172.21.15.110:41032) 2024-05-04T17:39:57.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb290f27640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb28c06f7c0 0x7fb28c14a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51066/0 (socket says 172.21.15.110:51066) 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28b7fe640 1 -- 172.21.15.110:0/1185616245 learned_addr learned my addr 172.21.15.110:0/1185616245 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3737785354 0 0) 0x7fb28c142860 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb274003520 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196649640 0 0) 0x7fb28c13e900 con 0x7fb28c142f50 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb28c142860 con 0x7fb28c142f50 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1700799369 0 0) 0x7fb274003520 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 >> v1:172.21.15.110:6790/0 conn(0x7fb28c142f50 legacy=0x7fb28c06f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 >> v1:172.21.15.136:6789/0 conn(0x7fb28c142480 legacy=0x7fb28c13e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28c13fca0 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb284001990 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb284001c70 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.362+0000 7fb28affd640 1 -- 172.21.15.110:0/1185616245 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb284002040 con 0x7fb28c06f7c0 2024-05-04T17:39:57.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.363+0000 7fb29298a640 1 -- 172.21.15.110:0/1185616245 >> v1:172.21.15.110:6789/0 conn(0x7fb28c06f7c0 legacy=0x7fb28c14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.363+0000 7fb29298a640 1 -- 172.21.15.110:0/1185616245 shutdown_connections 2024-05-04T17:39:57.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.363+0000 7fb29298a640 1 -- 172.21.15.110:0/1185616245 >> 172.21.15.110:0/1185616245 conn(0x7fb28c0b43a0 msgr2=0x7fb28c0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.363+0000 7fb29298a640 1 -- 172.21.15.110:0/1185616245 shutdown_connections 2024-05-04T17:39:57.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.363+0000 7fb29298a640 1 -- 172.21.15.110:0/1185616245 wait complete. 2024-05-04T17:39:57.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.364+0000 7fb29298a640 1 Processor -- start 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.364+0000 7fb29298a640 1 -- start start 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb29298a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c1349d0 con 0x7fb28c06f7c0 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb29298a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c13bd10 con 0x7fb28c142480 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb29298a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28c13bff0 con 0x7fb28c13a520 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb28bfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb28c06f7c0 0x7fb28c139e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51074/0 (socket says 172.21.15.110:51074) 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb290f27640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fb28c13a520 0x7fb28c1342c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41038/0 (socket says 172.21.15.110:41038) 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb28bfff640 1 -- 172.21.15.110:0/2725711918 learned_addr learned my addr 172.21.15.110:0/2725711918 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831801437 0 0) 0x7fb28c13bff0 con 0x7fb28c13a520 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb278003520 con 0x7fb28c13a520 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087038601 0 0) 0x7fb28c1349d0 con 0x7fb28c06f7c0 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb28c13bff0 con 0x7fb28c06f7c0 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157364869 0 0) 0x7fb28c13bd10 con 0x7fb28c142480 2024-05-04T17:39:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb28c1349d0 con 0x7fb28c142480 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1322799207 0 0) 0x7fb278003520 con 0x7fb28c13a520 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb28c13bd10 con 0x7fb28c13a520 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb284001fc0 con 0x7fb28c13a520 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2782031353 0 0) 0x7fb28c13bff0 con 0x7fb28c06f7c0 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb278003520 con 0x7fb28c06f7c0 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb27c0018b0 con 0x7fb28c06f7c0 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3902651916 0 0) 0x7fb28c1349d0 con 0x7fb28c142480 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb28c13bff0 con 0x7fb28c142480 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb2800017b0 con 0x7fb28c142480 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 957023374 0 0) 0x7fb28c13bd10 con 0x7fb28c13a520 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 >> v1:172.21.15.136:6789/0 conn(0x7fb28c142480 legacy=0x7fb28c133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 >> v1:172.21.15.110:6789/0 conn(0x7fb28c06f7c0 legacy=0x7fb28c139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28c1371c0 con 0x7fb28c13a520 2024-05-04T17:39:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb28c1e3350 con 0x7fb28c13a520 2024-05-04T17:39:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb28c1e3700 con 0x7fb28c13a520 2024-05-04T17:39:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb284001c70 con 0x7fb28c13a520 2024-05-04T17:39:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.365+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7fb2840050e0 con 0x7fb28c13a520 2024-05-04T17:39:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.366+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26c005080 con 0x7fb28c13a520 2024-05-04T17:39:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.367+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7fb28401b250 con 0x7fb28c13a520 2024-05-04T17:39:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.367+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7fb2840864d0 con 0x7fb28c13a520 2024-05-04T17:39:57.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.371+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7fb2840551f0 con 0x7fb28c13a520 2024-05-04T17:39:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.546+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fb26c000b90 con 0x7fb27806ca20 2024-05-04T17:39:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.546+0000 7fb288ff9640 1 -- 172.21.15.110:0/2725711918 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7fb26c000b90 con 0x7fb27806ca20 2024-05-04T17:39:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 3 to deep-scrub 2024-05-04T17:39:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.548+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 >> v1:172.21.15.110:6816/783516658 conn(0x7fb27806ca20 legacy=0x7fb27806eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.548+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 >> v1:172.21.15.110:6790/0 conn(0x7fb28c13a520 legacy=0x7fb28c1342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.548+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 shutdown_connections 2024-05-04T17:39:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.548+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 >> 172.21.15.110:0/2725711918 conn(0x7fb28c0b43a0 msgr2=0x7fb28c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.549+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 shutdown_connections 2024-05-04T17:39:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.549+0000 7fb29298a640 1 -- 172.21.15.110:0/2725711918 wait complete. 2024-05-04T17:39:57.561 INFO:tasks.ceph:Scrubbing osd.4 2024-05-04T17:39:57.561 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.660+0000 7f5d9f0c1640 1 Processor -- start 2024-05-04T17:39:57.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.660+0000 7f5d9f0c1640 1 -- start start 2024-05-04T17:39:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d9f0c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d98073560 con 0x7f5d98148d50 2024-05-04T17:39:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d9f0c1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d98072870 con 0x7f5d98071570 2024-05-04T17:39:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d9f0c1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d9813f440 con 0x7f5d9806f980 2024-05-04T17:39:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d9ce5d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f5d9806f980 0x7f5d9813e920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41042/0 (socket says 172.21.15.110:41042) 2024-05-04T17:39:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d9d65e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5d98148d50 0x7f5d9814b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51088/0 (socket says 172.21.15.110:51088) 2024-05-04T17:39:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d9ce5d640 1 -- 172.21.15.110:0/2100380111 learned_addr learned my addr 172.21.15.110:0/2100380111 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4245062880 0 0) 0x7f5d98073560 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d80003520 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512376413 0 0) 0x7f5d9813f440 con 0x7f5d9806f980 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d98073560 con 0x7f5d9806f980 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 570148683 0 0) 0x7f5d80003520 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 >> v1:172.21.15.110:6790/0 conn(0x7f5d9806f980 legacy=0x7f5d9813e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 >> v1:172.21.15.136:6789/0 conn(0x7f5d98071570 legacy=0x7f5d980719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d980730b0 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f5d900018c0 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d90001b20 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.661+0000 7f5d977fe640 1 -- 172.21.15.110:0/2100380111 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f5d90001f30 con 0x7f5d98148d50 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.662+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/2100380111 >> v1:172.21.15.110:6789/0 conn(0x7f5d98148d50 legacy=0x7f5d9814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.662+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/2100380111 shutdown_connections 2024-05-04T17:39:57.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.662+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/2100380111 >> 172.21.15.110:0/2100380111 conn(0x7f5d980b0e30 msgr2=0x7f5d980b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.662+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/2100380111 shutdown_connections 2024-05-04T17:39:57.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.662+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/2100380111 wait complete. 2024-05-04T17:39:57.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.663+0000 7f5d9f0c1640 1 Processor -- start 2024-05-04T17:39:57.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.663+0000 7f5d9f0c1640 1 -- start start 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9f0c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d98148910 con 0x7f5d9806f980 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9f0c1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d98136a00 con 0x7f5d98071570 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9f0c1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d9813e2d0 con 0x7f5d98148d50 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9ce5d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5d9806f980 0x7f5d98141b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51104/0 (socket says 172.21.15.110:51104) 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9ce5d640 1 -- 172.21.15.110:0/1987968875 learned_addr learned my addr 172.21.15.110:0/1987968875 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466995719 0 0) 0x7f5d98148910 con 0x7f5d9806f980 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d84003520 con 0x7f5d9806f980 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2248226064 0 0) 0x7f5d9813e2d0 con 0x7f5d98148d50 2024-05-04T17:39:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d98148910 con 0x7f5d98148d50 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131287821 0 0) 0x7f5d98136a00 con 0x7f5d98071570 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d9813e2d0 con 0x7f5d98071570 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1154034248 0 0) 0x7f5d84003520 con 0x7f5d9806f980 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d98136a00 con 0x7f5d9806f980 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f5d880012f0 con 0x7f5d9806f980 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 598793708 0 0) 0x7f5d98148910 con 0x7f5d98148d50 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d84003520 con 0x7f5d98148d50 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f5d90001e70 con 0x7f5d98148d50 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 618525451 0 0) 0x7f5d9813e2d0 con 0x7f5d98071570 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d98148910 con 0x7f5d98071570 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f5d8c001830 con 0x7f5d98071570 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3961390643 0 0) 0x7f5d98136a00 con 0x7f5d9806f980 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 >> v1:172.21.15.110:6790/0 conn(0x7f5d98148d50 legacy=0x7f5d981e2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 >> v1:172.21.15.136:6789/0 conn(0x7f5d98071570 legacy=0x7f5d981dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d98135110 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9813d9d0 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.664+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d98142480 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.665+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d880015e0 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.665+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f5d88004c30 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.665+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5d98062e90 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.666+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f5d8801ada0 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.666+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f5d88086170 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.666+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.136:6804/2025414972 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d840784d0 con 0x7f5d84074860 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.666+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7f5d88086640 con 0x7f5d9806f980 2024-05-04T17:39:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.668+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== osd.4 v1:172.21.15.136:6804/2025414972 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5d840784d0 con 0x7f5d84074860 2024-05-04T17:39:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.681+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 --> v1:172.21.15.136:6804/2025414972 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5d98130080 con 0x7f5d84074860 2024-05-04T17:39:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.681+0000 7f5d957fa640 1 -- 172.21.15.110:0/1987968875 <== osd.4 v1:172.21.15.136:6804/2025414972 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f5d98130080 con 0x7f5d84074860 2024-05-04T17:39:57.683 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:57.683 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:57.683 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 >> v1:172.21.15.136:6804/2025414972 conn(0x7f5d84074860 legacy=0x7f5d84076c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 >> v1:172.21.15.110:6816/783516658 conn(0x7f5d8406ca70 legacy=0x7f5d8406ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 >> v1:172.21.15.110:6789/0 conn(0x7f5d9806f980 legacy=0x7f5d98141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9d65e640 1 -- 172.21.15.110:0/1987968875 reap_dead start 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 shutdown_connections 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 >> 172.21.15.110:0/1987968875 conn(0x7f5d980b0e30 msgr2=0x7f5d98142ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 shutdown_connections 2024-05-04T17:39:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.682+0000 7f5d9f0c1640 1 -- 172.21.15.110:0/1987968875 wait complete. 2024-05-04T17:39:57.694 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-04T17:39:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.790+0000 7f75e5d37640 1 Processor -- start 2024-05-04T17:39:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.790+0000 7f75e5d37640 1 -- start start 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75e5d37640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e0146460 con 0x7f75e0146710 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75e5d37640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e0073120 con 0x7f75e0146080 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75e5d37640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e0071400 con 0x7f75e0142340 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75deffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f75e0146710 0x7f75e0141c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51120/0 (socket says 172.21.15.110:51120) 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75deffd640 1 -- 172.21.15.110:0/3256278202 learned_addr learned my addr 172.21.15.110:0/3256278202 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75dffff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f75e0142340 0x7f75e014e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41060/0 (socket says 172.21.15.110:41060) 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289739704 0 0) 0x7f75e0146460 con 0x7f75e0146710 2024-05-04T17:39:57.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75c8003520 con 0x7f75e0146710 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867062179 0 0) 0x7f75e0071400 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.791+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e0146460 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055325395 0 0) 0x7f75e0073120 con 0x7f75e0146080 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e0071400 con 0x7f75e0146080 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 563963879 0 0) 0x7f75e0146460 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 >> v1:172.21.15.136:6789/0 conn(0x7f75e0146080 legacy=0x7f75e006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 >> v1:172.21.15.110:6789/0 conn(0x7f75e0146710 legacy=0x7f75e0141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e0071f60 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f75d8001970 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75d8001c60 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75de7fc640 1 -- 172.21.15.110:0/3256278202 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f75d8002070 con 0x7f75e0142340 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75e5d37640 1 -- 172.21.15.110:0/3256278202 >> v1:172.21.15.110:6790/0 conn(0x7f75e0142340 legacy=0x7f75e014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75e5d37640 1 -- 172.21.15.110:0/3256278202 shutdown_connections 2024-05-04T17:39:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75e5d37640 1 -- 172.21.15.110:0/3256278202 >> 172.21.15.110:0/3256278202 conn(0x7f75e00b8920 msgr2=0x7f75e00b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75e5d37640 1 -- 172.21.15.110:0/3256278202 shutdown_connections 2024-05-04T17:39:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.792+0000 7f75e5d37640 1 -- 172.21.15.110:0/3256278202 wait complete. 2024-05-04T17:39:57.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.793+0000 7f75e5d37640 1 Processor -- start 2024-05-04T17:39:57.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75e5d37640 1 -- start start 2024-05-04T17:39:57.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75e5d37640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e00b0f00 con 0x7f75e0142340 2024-05-04T17:39:57.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75e5d37640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e00afe40 con 0x7f75e0146080 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75e5d37640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e013afb0 con 0x7f75e0146710 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75df7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f75e0142340 0x7f75e00b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51132/0 (socket says 172.21.15.110:51132) 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75df7fe640 1 -- 172.21.15.110:0/1681900939 learned_addr learned my addr 172.21.15.110:0/1681900939 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.794+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361996013 0 0) 0x7f75e013afb0 con 0x7f75e0146710 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75cc003520 con 0x7f75e0146710 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2902536420 0 0) 0x7f75e00b0f00 con 0x7f75e0142340 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e013afb0 con 0x7f75e0142340 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510842937 0 0) 0x7f75e00afe40 con 0x7f75e0146080 2024-05-04T17:39:57.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e00b0f00 con 0x7f75e0146080 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4251140782 0 0) 0x7f75cc003520 con 0x7f75e0146710 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75e00afe40 con 0x7f75e0146710 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f75d8001930 con 0x7f75e0146710 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3288431819 0 0) 0x7f75e013afb0 con 0x7f75e0142340 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75cc003520 con 0x7f75e0142340 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f75d00018f0 con 0x7f75e0142340 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2597313069 0 0) 0x7f75e00b0f00 con 0x7f75e0146080 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75e013afb0 con 0x7f75e0146080 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f75d4001790 con 0x7f75e0146080 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2340040792 0 0) 0x7f75e00afe40 con 0x7f75e0146710 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 >> v1:172.21.15.136:6789/0 conn(0x7f75e0146080 legacy=0x7f75e013a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 >> v1:172.21.15.110:6789/0 conn(0x7f75e0142340 legacy=0x7f75e00b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e013b180 con 0x7f75e0146710 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f75e0145590 con 0x7f75e0146710 2024-05-04T17:39:57.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f75e01db3a0 con 0x7f75e0146710 2024-05-04T17:39:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f75d8001c60 con 0x7f75e0146710 2024-05-04T17:39:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.795+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f75d80050d0 con 0x7f75e0146710 2024-05-04T17:39:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.796+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c0005080 con 0x7f75e0146710 2024-05-04T17:39:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.797+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f75d801b240 con 0x7f75e0146710 2024-05-04T17:39:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.797+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f75d8086490 con 0x7f75e0146710 2024-05-04T17:39:57.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.802+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f75d80552b0 con 0x7f75e0146710 2024-05-04T17:39:57.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.989+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f75c0000b90 con 0x7f75cc06ca20 2024-05-04T17:39:57.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.989+0000 7f75c7fff640 1 -- 172.21.15.110:0/1681900939 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f75c0000b90 con 0x7f75cc06ca20 2024-05-04T17:39:57.990 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 4 to deep-scrub 2024-05-04T17:39:57.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.991+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 >> v1:172.21.15.110:6816/783516658 conn(0x7f75cc06ca20 legacy=0x7f75cc06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.991+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 >> v1:172.21.15.110:6790/0 conn(0x7f75e0146710 legacy=0x7f75e01dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:57.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.991+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 shutdown_connections 2024-05-04T17:39:57.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.991+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 >> 172.21.15.110:0/1681900939 conn(0x7f75e00b8920 msgr2=0x7f75e013dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:57.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.991+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 shutdown_connections 2024-05-04T17:39:57.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:57.991+0000 7f75e5d37640 1 -- 172.21.15.110:0/1681900939 wait complete. 2024-05-04T17:39:58.005 INFO:tasks.ceph:Scrubbing osd.5 2024-05-04T17:39:58.005 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:58.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.121+0000 7f731944b640 1 Processor -- start 2024-05-04T17:39:58.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.121+0000 7f731944b640 1 -- start start 2024-05-04T17:39:58.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f731944b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73140a9b30 con 0x7f73140aa1e0 2024-05-04T17:39:58.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f731944b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7314070cb0 con 0x7f7314070820 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f731944b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73140ab460 con 0x7f73140a9750 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f7312ffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f73140a9750 0x7f7314140d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41078/0 (socket says 172.21.15.110:41078) 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f7312ffd640 1 -- 172.21.15.110:0/162393231 learned_addr learned my addr 172.21.15.110:0/162393231 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896481539 0 0) 0x7f73140ab460 con 0x7f73140a9750 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72fc003520 con 0x7f73140a9750 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.122+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067344142 0 0) 0x7f73140a9b30 con 0x7f73140aa1e0 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73140ab460 con 0x7f73140aa1e0 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204986909 0 0) 0x7f730c001490 con 0x7f7314070820 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73140a9b30 con 0x7f7314070820 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2792119378 0 0) 0x7f73140ab460 con 0x7f73140aa1e0 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 >> v1:172.21.15.110:6790/0 conn(0x7f73140a9750 legacy=0x7f7314140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 >> v1:172.21.15.136:6789/0 conn(0x7f7314070820 legacy=0x7f731414bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73140733c0 con 0x7f73140aa1e0 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7308001590 con 0x7f73140aa1e0 2024-05-04T17:39:58.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7308001840 con 0x7f73140aa1e0 2024-05-04T17:39:58.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.123+0000 7f7311ffb640 1 -- 172.21.15.110:0/162393231 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7308001c10 con 0x7f73140aa1e0 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.124+0000 7f731944b640 1 -- 172.21.15.110:0/162393231 >> v1:172.21.15.110:6789/0 conn(0x7f73140aa1e0 legacy=0x7f731413a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.124+0000 7f731944b640 1 -- 172.21.15.110:0/162393231 shutdown_connections 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.124+0000 7f731944b640 1 -- 172.21.15.110:0/162393231 >> 172.21.15.110:0/162393231 conn(0x7f73140b6680 msgr2=0x7f73140b6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.124+0000 7f731944b640 1 -- 172.21.15.110:0/162393231 shutdown_connections 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.124+0000 7f731944b640 1 -- 172.21.15.110:0/162393231 wait complete. 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.125+0000 7f731944b640 1 Processor -- start 2024-05-04T17:39:58.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.125+0000 7f731944b640 1 -- start start 2024-05-04T17:39:58.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f731944b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73141d8540 con 0x7f73140aa1e0 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f731944b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73140ae060 con 0x7f7314070820 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f731944b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73141d7630 con 0x7f73140a9750 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f73137fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f73140aa1e0 0x7f73141348f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51146/0 (socket says 172.21.15.110:51146) 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f73137fe640 1 -- 172.21.15.110:0/163196392 learned_addr learned my addr 172.21.15.110:0/163196392 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f73127fc640 1 --1- 172.21.15.110:0/163196392 >> v1:172.21.15.110:6790/0 conn(0x7f73140a9750 0x7f73141d7e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41088/0 (socket says 172.21.15.110:41088) 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900851650 0 0) 0x7f73141d7630 con 0x7f73140a9750 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7300003520 con 0x7f73140a9750 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730758487 0 0) 0x7f73141d8540 con 0x7f73140aa1e0 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73141d7630 con 0x7f73140aa1e0 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099146467 0 0) 0x7f73140ae060 con 0x7f7314070820 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.126+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73141d8540 con 0x7f7314070820 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3293863890 0 0) 0x7f73141d8540 con 0x7f7314070820 2024-05-04T17:39:58.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73140ae060 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7304001910 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1006906088 0 0) 0x7f73141d7630 con 0x7f73140aa1e0 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73141d8540 con 0x7f73140aa1e0 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f730c001d40 con 0x7f73140aa1e0 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2300331682 0 0) 0x7f7300003520 con 0x7f73140a9750 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73141d7630 con 0x7f73140a9750 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7308001550 con 0x7f73140a9750 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 414361282 0 0) 0x7f73140ae060 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 >> v1:172.21.15.110:6790/0 conn(0x7f73140a9750 legacy=0x7f73141d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 >> v1:172.21.15.110:6789/0 conn(0x7f73140aa1e0 legacy=0x7f73141348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73141d7910 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73140abe10 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7314135000 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73040047e0 con 0x7f7314070820 2024-05-04T17:39:58.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.127+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7304004b30 con 0x7f7314070820 2024-05-04T17:39:58.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.128+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f72f4000f80 con 0x7f7314070820 2024-05-04T17:39:58.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.129+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f730401aca0 con 0x7f7314070820 2024-05-04T17:39:58.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.129+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f73040861f0 con 0x7f7314070820 2024-05-04T17:39:58.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.129+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6800/2872378682 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7300078540 con 0x7f7300074970 2024-05-04T17:39:58.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.129+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7f7304086600 con 0x7f7314070820 2024-05-04T17:39:58.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.130+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== osd.5 v1:172.21.15.136:6800/2872378682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7300078540 con 0x7f7300074970 2024-05-04T17:39:58.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.147+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 --> v1:172.21.15.136:6800/2872378682 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f72f4002cb0 con 0x7f7300074970 2024-05-04T17:39:58.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f72fb7fe640 1 -- 172.21.15.110:0/163196392 <== osd.5 v1:172.21.15.136:6800/2872378682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f72f4002cb0 con 0x7f7300074970 2024-05-04T17:39:58.149 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:58.149 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:58.150 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:58.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 >> v1:172.21.15.136:6800/2872378682 conn(0x7f7300074970 legacy=0x7f7300076d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 >> v1:172.21.15.110:6816/783516658 conn(0x7f730006cb60 legacy=0x7f730006f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 >> v1:172.21.15.136:6789/0 conn(0x7f7314070820 legacy=0x7f731406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f73137fe640 1 -- 172.21.15.110:0/163196392 reap_dead start 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 shutdown_connections 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 >> 172.21.15.110:0/163196392 conn(0x7f73140b6680 msgr2=0x7f7314139700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 shutdown_connections 2024-05-04T17:39:58.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.148+0000 7f731944b640 1 -- 172.21.15.110:0/163196392 wait complete. 2024-05-04T17:39:58.163 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-04T17:39:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f0269ce0640 1 Processor -- start 2024-05-04T17:39:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f0269ce0640 1 -- start start 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f0269ce0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0264146460 con 0x7f0264146080 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f0269ce0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0264073120 con 0x7f0264146710 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f0269ce0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0264071400 con 0x7f0264142340 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f02637fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0264146080 0x7f026406e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51156/0 (socket says 172.21.15.110:51156) 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f02637fe640 1 -- 172.21.15.110:0/1880760524 learned_addr learned my addr 172.21.15.110:0/1880760524 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.274+0000 7f0263fff640 1 --1- 172.21.15.110:0/1880760524 >> v1:172.21.15.110:6790/0 conn(0x7f0264142340 0x7f026414e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41090/0 (socket says 172.21.15.110:41090) 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 32498778 0 0) 0x7f0264071400 con 0x7f0264142340 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0250003520 con 0x7f0264142340 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349514598 0 0) 0x7f0264146460 con 0x7f0264146080 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0264071400 con 0x7f0264146080 2024-05-04T17:39:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 500782399 0 0) 0x7f0250003520 con 0x7f0264142340 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 >> v1:172.21.15.136:6789/0 conn(0x7f0264146710 legacy=0x7f0264141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 >> v1:172.21.15.110:6789/0 conn(0x7f0264146080 legacy=0x7f026406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0264071f60 con 0x7f0264142340 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f025c001990 con 0x7f0264142340 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f025c001c00 con 0x7f0264142340 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.275+0000 7f0262ffd640 1 -- 172.21.15.110:0/1880760524 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f025c002010 con 0x7f0264142340 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.276+0000 7f0269ce0640 1 -- 172.21.15.110:0/1880760524 >> v1:172.21.15.110:6790/0 conn(0x7f0264142340 legacy=0x7f026414e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.276+0000 7f0269ce0640 1 -- 172.21.15.110:0/1880760524 shutdown_connections 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.276+0000 7f0269ce0640 1 -- 172.21.15.110:0/1880760524 >> 172.21.15.110:0/1880760524 conn(0x7f02640b8920 msgr2=0x7f02640b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.276+0000 7f0269ce0640 1 -- 172.21.15.110:0/1880760524 shutdown_connections 2024-05-04T17:39:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.276+0000 7f0269ce0640 1 -- 172.21.15.110:0/1880760524 wait complete. 2024-05-04T17:39:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0269ce0640 1 Processor -- start 2024-05-04T17:39:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0269ce0640 1 -- start start 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0269ce0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02640b2d80 con 0x7f0264142340 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0269ce0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02640b4750 con 0x7f0264146710 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0269ce0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02640b42d0 con 0x7f0264146080 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f02637fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0264142340 0x7f02640b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51160/0 (socket says 172.21.15.110:51160) 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f02637fe640 1 -- 172.21.15.110:0/1902349117 learned_addr learned my addr 172.21.15.110:0/1902349117 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574540701 0 0) 0x7f02640b2d80 con 0x7f0264142340 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0244003520 con 0x7f0264142340 2024-05-04T17:39:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296009709 0 0) 0x7f02640b42d0 con 0x7f0264146080 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.277+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02640b2d80 con 0x7f0264146080 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165264186 0 0) 0x7f02640b4750 con 0x7f0264146710 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02640b42d0 con 0x7f0264146710 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1025940021 0 0) 0x7f0244003520 con 0x7f0264142340 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02640b4750 con 0x7f0264142340 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2131043984 0 0) 0x7f02640b2d80 con 0x7f0264146080 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0244003520 con 0x7f0264146080 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3365958762 0 0) 0x7f02640b42d0 con 0x7f0264146710 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02640b2d80 con 0x7f0264146710 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f02540019b0 con 0x7f0264142340 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f024c001710 con 0x7f0264146080 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f025c001db0 con 0x7f0264146710 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3202070982 0 0) 0x7f02640b4750 con 0x7f0264142340 2024-05-04T17:39:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 >> v1:172.21.15.110:6790/0 conn(0x7f0264146080 legacy=0x7f026413ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 >> v1:172.21.15.136:6789/0 conn(0x7f0264146710 legacy=0x7f02641dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026413c150 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f026413c3a0 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f02640b7e80 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0254001cc0 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.278+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0254004c30 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.279+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0248005080 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.279+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f025401ada0 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.280+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f0254086340 con 0x7f0264142340 2024-05-04T17:39:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.284+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f0254055080 con 0x7f0264142340 2024-05-04T17:39:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.460+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f0248000b90 con 0x7f024406ca00 2024-05-04T17:39:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.460+0000 7f0260ff9640 1 -- 172.21.15.110:0/1902349117 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f0248000b90 con 0x7f024406ca00 2024-05-04T17:39:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 5 to deep-scrub 2024-05-04T17:39:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.462+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 >> v1:172.21.15.110:6816/783516658 conn(0x7f024406ca00 legacy=0x7f024406eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.462+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 >> v1:172.21.15.110:6789/0 conn(0x7f0264142340 legacy=0x7f02640b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.462+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 shutdown_connections 2024-05-04T17:39:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.462+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 >> 172.21.15.110:0/1902349117 conn(0x7f02640b8920 msgr2=0x7f026413db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.462+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 shutdown_connections 2024-05-04T17:39:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.462+0000 7f0269ce0640 1 -- 172.21.15.110:0/1902349117 wait complete. 2024-05-04T17:39:58.475 INFO:tasks.ceph:Scrubbing osd.6 2024-05-04T17:39:58.475 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:58.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.573+0000 7f46ad350640 1 Processor -- start 2024-05-04T17:39:58.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.573+0000 7f46ad350640 1 -- start start 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46ad350640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a81463e0 con 0x7f46a81423c0 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46ad350640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a8146c30 con 0x7f46a81467d0 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46ad350640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a806fd30 con 0x7f46a8146000 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a6ffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f46a8146000 0x7f46a806f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41112/0 (socket says 172.21.15.110:41112) 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f46a81423c0 0x7f46a814e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51170/0 (socket says 172.21.15.110:51170) 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a6ffd640 1 -- 172.21.15.110:0/1920137574 learned_addr learned my addr 172.21.15.110:0/1920137574 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2960758534 0 0) 0x7f46a806fd30 con 0x7f46a8146000 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4690003520 con 0x7f46a8146000 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479768131 0 0) 0x7f46a81463e0 con 0x7f46a81423c0 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a806fd30 con 0x7f46a81423c0 2024-05-04T17:39:58.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a67fc640 1 -- 172.21.15.110:0/1920137574 >> v1:172.21.15.136:6789/0 conn(0x7f46a81467d0 legacy=0x7f46a8141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a67fc640 1 -- 172.21.15.110:0/1920137574 >> v1:172.21.15.136:6789/0 conn(0x7f46a81467d0 legacy=0x7f46a8141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a67fc640 1 --1- 172.21.15.110:0/1920137574 >> v1:172.21.15.136:6789/0 conn(0x7f46a81467d0 0x7f46a8141cb0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3574942554 0 0) 0x7f4690003520 con 0x7f46a8146000 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 >> v1:172.21.15.136:6789/0 conn(0x7f46a81467d0 legacy=0x7f46a8141cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 >> v1:172.21.15.110:6789/0 conn(0x7f46a81423c0 legacy=0x7f46a814e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a8071660 con 0x7f46a8146000 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.574+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4698001990 con 0x7f46a8146000 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4698001c30 con 0x7f46a8146000 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46a5ffb640 1 -- 172.21.15.110:0/1920137574 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4698002040 con 0x7f46a8146000 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46ad350640 1 -- 172.21.15.110:0/1920137574 >> v1:172.21.15.110:6790/0 conn(0x7f46a8146000 legacy=0x7f46a806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46ad350640 1 -- 172.21.15.110:0/1920137574 shutdown_connections 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46ad350640 1 -- 172.21.15.110:0/1920137574 >> 172.21.15.110:0/1920137574 conn(0x7f46a80b51d0 msgr2=0x7f46a80b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46ad350640 1 -- 172.21.15.110:0/1920137574 shutdown_connections 2024-05-04T17:39:58.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.575+0000 7f46ad350640 1 -- 172.21.15.110:0/1920137574 wait complete. 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.576+0000 7f46ad350640 1 Processor -- start 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.576+0000 7f46ad350640 1 -- start start 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f46ad350640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a81d8450 con 0x7f46a8146000 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f46ad350640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a813c1b0 con 0x7f46a81423c0 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f46ad350640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a81d7300 con 0x7f46a81467d0 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f46a67fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f46a8146000 0x7f46a81d7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51174/0 (socket says 172.21.15.110:51174) 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f46a77fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f46a81467d0 0x7f46a81e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41116/0 (socket says 172.21.15.110:41116) 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f46a67fc640 1 -- 172.21.15.110:0/1513989360 learned_addr learned my addr 172.21.15.110:0/1513989360 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831693773 0 0) 0x7f46a81d7300 con 0x7f46a81467d0 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4694003520 con 0x7f46a81467d0 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765398662 0 0) 0x7f46a81d8450 con 0x7f46a8146000 2024-05-04T17:39:58.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a81d7300 con 0x7f46a8146000 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956915660 0 0) 0x7f46a813c1b0 con 0x7f46a81423c0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a81d8450 con 0x7f46a81423c0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2242793127 0 0) 0x7f4694003520 con 0x7f46a81467d0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46a813c1b0 con 0x7f46a81467d0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f46a0001890 con 0x7f46a81467d0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2247010160 0 0) 0x7f46a81d7300 con 0x7f46a8146000 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4694003520 con 0x7f46a8146000 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2838584983 0 0) 0x7f46a81d8450 con 0x7f46a81423c0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46a81d7300 con 0x7f46a81423c0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.577+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4698001890 con 0x7f46a81423c0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4142888312 0 0) 0x7f46a813c1b0 con 0x7f46a81467d0 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 >> v1:172.21.15.136:6789/0 conn(0x7f46a81423c0 legacy=0x7f46a8139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 >> v1:172.21.15.110:6789/0 conn(0x7f46a8146000 legacy=0x7f46a81d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a81d74d0 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46a813ca20 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f46a8136910 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46a00047e0 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f46a0004b30 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.578+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.110:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4688000f80 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.579+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f46a001aca0 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.579+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f46a0085ff0 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.579+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.136:6812/239937373 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46940784c0 con 0x7f46940748f0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.579+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7f46a0086460 con 0x7f46a81467d0 2024-05-04T17:39:58.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.581+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== osd.6 v1:172.21.15.136:6812/239937373 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f46940784c0 con 0x7f46940748f0 2024-05-04T17:39:58.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.593+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 --> v1:172.21.15.136:6812/239937373 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4688002cb0 con 0x7f46940748f0 2024-05-04T17:39:58.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f468f7fe640 1 -- 172.21.15.110:0/1513989360 <== osd.6 v1:172.21.15.136:6812/239937373 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4688002cb0 con 0x7f46940748f0 2024-05-04T17:39:58.595 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:58.595 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:58.595 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:58.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 >> v1:172.21.15.136:6812/239937373 conn(0x7f46940748f0 legacy=0x7f4694076d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 >> v1:172.21.15.110:6816/783516658 conn(0x7f469406cae0 legacy=0x7f469406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 >> v1:172.21.15.110:6790/0 conn(0x7f46a81467d0 legacy=0x7f46a81e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46a77fe640 1 -- 172.21.15.110:0/1513989360 reap_dead start 2024-05-04T17:39:58.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 shutdown_connections 2024-05-04T17:39:58.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 >> 172.21.15.110:0/1513989360 conn(0x7f46a80b51d0 msgr2=0x7f46a8135f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 shutdown_connections 2024-05-04T17:39:58.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.594+0000 7f46ad350640 1 -- 172.21.15.110:0/1513989360 wait complete. 2024-05-04T17:39:58.606 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-04T17:39:58.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.707+0000 7f7f084eb640 1 Processor -- start 2024-05-04T17:39:58.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.707+0000 7f7f084eb640 1 -- start start 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.708+0000 7f7f084eb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f00146460 con 0x7f7f00146710 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.708+0000 7f7f084eb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f00073120 con 0x7f7f00146080 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.708+0000 7f7f084eb640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f00071400 con 0x7f7f00142340 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.708+0000 7f7f05a86640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7f00146710 0x7f7f00141c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51180/0 (socket says 172.21.15.110:51180) 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.708+0000 7f7f05a86640 1 -- 172.21.15.110:0/3861325942 learned_addr learned my addr 172.21.15.110:0/3861325942 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.708+0000 7f7f06a88640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f7f00142340 0x7f7f0014e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41122/0 (socket says 172.21.15.110:41122) 2024-05-04T17:39:58.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509382142 0 0) 0x7f7f00146460 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eec003520 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064771323 0 0) 0x7f7f00071400 con 0x7f7f00142340 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f00146460 con 0x7f7f00142340 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905993486 0 0) 0x7f7f00073120 con 0x7f7f00146080 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f00071400 con 0x7f7f00146080 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2770164799 0 0) 0x7f7eec003520 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 >> v1:172.21.15.110:6790/0 conn(0x7f7f00142340 legacy=0x7f7f0014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 >> v1:172.21.15.136:6789/0 conn(0x7f7f00146080 legacy=0x7f7f0006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f00071f60 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7ef0001610 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ef00018c0 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.709+0000 7f7f05285640 1 -- 172.21.15.110:0/3861325942 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7ef0001c90 con 0x7f7f00146710 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.710+0000 7f7f084eb640 1 -- 172.21.15.110:0/3861325942 >> v1:172.21.15.110:6789/0 conn(0x7f7f00146710 legacy=0x7f7f00141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.710+0000 7f7f084eb640 1 -- 172.21.15.110:0/3861325942 shutdown_connections 2024-05-04T17:39:58.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.710+0000 7f7f084eb640 1 -- 172.21.15.110:0/3861325942 >> 172.21.15.110:0/3861325942 conn(0x7f7f000b8920 msgr2=0x7f7f000b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.710+0000 7f7f084eb640 1 -- 172.21.15.110:0/3861325942 shutdown_connections 2024-05-04T17:39:58.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.710+0000 7f7f084eb640 1 -- 172.21.15.110:0/3861325942 wait complete. 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f084eb640 1 Processor -- start 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f084eb640 1 -- start start 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f084eb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f000b2440 con 0x7f7f00146710 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f084eb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f000b2720 con 0x7f7f00142340 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f084eb640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f000b1240 con 0x7f7f00146080 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f05a86640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f7f00146080 0x7f7f0013ad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41130/0 (socket says 172.21.15.110:41130) 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7f05a86640 1 -- 172.21.15.110:0/2610723645 learned_addr learned my addr 172.21.15.110:0/2610723645 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.711+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 646002497 0 0) 0x7f7f000b2440 con 0x7f7f00146710 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ee4003520 con 0x7f7f00146710 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125783705 0 0) 0x7f7f000b1240 con 0x7f7f00146080 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f000b2440 con 0x7f7f00146080 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854288321 0 0) 0x7f7f000b2720 con 0x7f7f00142340 2024-05-04T17:39:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f000b1240 con 0x7f7f00142340 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1693605581 0 0) 0x7f7ee4003520 con 0x7f7f00146710 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f000b2720 con 0x7f7f00146710 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3459364110 0 0) 0x7f7f000b2440 con 0x7f7f00146080 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ee4003520 con 0x7f7f00146080 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7ef8001c30 con 0x7f7f00146710 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3244086198 0 0) 0x7f7f000b1240 con 0x7f7f00142340 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f000b2440 con 0x7f7f00142340 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7ef0001c10 con 0x7f7f00146080 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7efc001830 con 0x7f7f00142340 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1499427269 0 0) 0x7f7f000b2720 con 0x7f7f00146710 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 >> v1:172.21.15.110:6790/0 conn(0x7f7f00146080 legacy=0x7f7f0013ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 >> v1:172.21.15.136:6789/0 conn(0x7f7f00142340 legacy=0x7f7f000b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f000b0bc0 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0013b460 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f001e3c20 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ef8004b60 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.712+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7ef8004eb0 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.713+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee8005080 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.713+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f7ef801b020 con 0x7f7f00146710 2024-05-04T17:39:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.714+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f7ef80864c0 con 0x7f7f00146710 2024-05-04T17:39:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.719+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f7ef80551a0 con 0x7f7f00146710 2024-05-04T17:39:58.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.903+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f7ee8000b90 con 0x7f7ee406cae0 2024-05-04T17:39:58.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.903+0000 7f7ef6ffd640 1 -- 172.21.15.110:0/2610723645 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (unknown 778718403 0 0) 0x7f7ee8000b90 con 0x7f7ee406cae0 2024-05-04T17:39:58.904 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 6 to deep-scrub 2024-05-04T17:39:58.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.905+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 >> v1:172.21.15.110:6816/783516658 conn(0x7f7ee406cae0 legacy=0x7f7ee406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.905+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 >> v1:172.21.15.110:6789/0 conn(0x7f7f00146710 legacy=0x7f7f001e2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:58.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.905+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 shutdown_connections 2024-05-04T17:39:58.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.905+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 >> 172.21.15.110:0/2610723645 conn(0x7f7f000b8920 msgr2=0x7f7f000b8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:58.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.905+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 shutdown_connections 2024-05-04T17:39:58.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:58.905+0000 7f7f084eb640 1 -- 172.21.15.110:0/2610723645 wait complete. 2024-05-04T17:39:58.920 INFO:tasks.ceph:Scrubbing osd.7 2024-05-04T17:39:58.920 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-04T17:39:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.024+0000 7f7178c4e640 1 Processor -- start 2024-05-04T17:39:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.024+0000 7f7178c4e640 1 -- start start 2024-05-04T17:39:59.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f7178c4e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7174142810 con 0x7f7174070650 2024-05-04T17:39:59.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f7178c4e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7174072980 con 0x7f7174142ea0 2024-05-04T17:39:59.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f7178c4e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7174071b90 con 0x7f7174142430 2024-05-04T17:39:59.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f7172d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7174070650 0x7f717414a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51208/0 (socket says 172.21.15.110:51208) 2024-05-04T17:39:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717259c640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f7174142430 0x7f717413e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41136/0 (socket says 172.21.15.110:41136) 2024-05-04T17:39:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f7172d9d640 1 -- 172.21.15.110:0/266067591 learned_addr learned my addr 172.21.15.110:0/266067591 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161647392 0 0) 0x7f7174071b90 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7160003520 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642092682 0 0) 0x7f7174142810 con 0x7f7174070650 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7174071b90 con 0x7f7174070650 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3977482988 0 0) 0x7f7160003520 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 >> v1:172.21.15.136:6789/0 conn(0x7f7174142ea0 legacy=0x7f717406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 >> v1:172.21.15.110:6789/0 conn(0x7f7174070650 legacy=0x7f717414a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717413fc50 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f715c001610 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f715c001870 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.025+0000 7f717159a640 1 -- 172.21.15.110:0/266067591 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f715c001c80 con 0x7f7174142430 2024-05-04T17:39:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.026+0000 7f7178c4e640 1 -- 172.21.15.110:0/266067591 >> v1:172.21.15.110:6790/0 conn(0x7f7174142430 legacy=0x7f717413e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.026+0000 7f7178c4e640 1 -- 172.21.15.110:0/266067591 shutdown_connections 2024-05-04T17:39:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.026+0000 7f7178c4e640 1 -- 172.21.15.110:0/266067591 >> 172.21.15.110:0/266067591 conn(0x7f71740b0e30 msgr2=0x7f71740b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.026+0000 7f7178c4e640 1 -- 172.21.15.110:0/266067591 shutdown_connections 2024-05-04T17:39:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.026+0000 7f7178c4e640 1 -- 172.21.15.110:0/266067591 wait complete. 2024-05-04T17:39:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.027+0000 7f7178c4e640 1 Processor -- start 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.027+0000 7f7178c4e640 1 -- start start 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f7178c4e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717413af10 con 0x7f7174142430 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f7178c4e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71740abad0 con 0x7f7174070650 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f7178c4e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71740ab760 con 0x7f7174142ea0 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f7171d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7174142430 0x7f717413a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51218/0 (socket says 172.21.15.110:51218) 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f7172d9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f7174142ea0 0x7f717413a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41148/0 (socket says 172.21.15.110:41148) 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f7171d9b640 1 -- 172.21.15.110:0/2817500009 learned_addr learned my addr 172.21.15.110:0/2817500009 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 178299412 0 0) 0x7f717413af10 con 0x7f7174142430 2024-05-04T17:39:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7154003520 con 0x7f7174142430 2024-05-04T17:39:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797465460 0 0) 0x7f71740ab760 con 0x7f7174142ea0 2024-05-04T17:39:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f717413af10 con 0x7f7174142ea0 2024-05-04T17:39:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1359277439 0 0) 0x7f71740abad0 con 0x7f7174070650 2024-05-04T17:39:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71740ab760 con 0x7f7174070650 2024-05-04T17:39:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2782822510 0 0) 0x7f7154003520 con 0x7f7174142430 2024-05-04T17:39:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71740abad0 con 0x7f7174142430 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f71640018e0 con 0x7f7174142430 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2047577233 0 0) 0x7f717413af10 con 0x7f7174142ea0 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7154003520 con 0x7f7174142ea0 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f716c001470 con 0x7f7174142ea0 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.028+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1250511440 0 0) 0x7f71740ab760 con 0x7f7174070650 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f717413af10 con 0x7f7174070650 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f715c001c00 con 0x7f7174070650 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3339067201 0 0) 0x7f71740abad0 con 0x7f7174142430 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 >> v1:172.21.15.110:6790/0 conn(0x7f7174142ea0 legacy=0x7f717413a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 >> v1:172.21.15.136:6789/0 conn(0x7f7174070650 legacy=0x7f71740a9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71740aa520 con 0x7f7174142430 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71740afe20 con 0x7f7174142430 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71740aca80 con 0x7f7174142430 2024-05-04T17:39:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7164001cb0 con 0x7f7174142430 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.029+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f7164003a60 con 0x7f7174142430 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.030+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.110:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7158000f80 con 0x7f7174142430 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.030+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f7164003a60 con 0x7f7174142430 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.031+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f7164086360 con 0x7f7174142430 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.031+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.136:6808/2884573928 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7154078550 con 0x7f71540748e0 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.031+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (unknown 816405262 0 0) 0x7f7164086770 con 0x7f7174142430 2024-05-04T17:39:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.032+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== osd.7 v1:172.21.15.136:6808/2884573928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7154078550 con 0x7f71540748e0 2024-05-04T17:39:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.045+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 --> v1:172.21.15.136:6808/2884573928 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7158002cb0 con 0x7f71540748e0 2024-05-04T17:39:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.045+0000 7f716affd640 1 -- 172.21.15.110:0/2817500009 <== osd.7 v1:172.21.15.136:6808/2884573928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f7158002cb0 con 0x7f71540748e0 2024-05-04T17:39:59.048 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-04T17:39:59.048 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T17:39:59.048 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-04T17:39:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 >> v1:172.21.15.136:6808/2884573928 conn(0x7f71540748e0 legacy=0x7f7154076d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 >> v1:172.21.15.110:6816/783516658 conn(0x7f715406cad0 legacy=0x7f715406ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 >> v1:172.21.15.110:6789/0 conn(0x7f7174142430 legacy=0x7f717413a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7172d9d640 1 -- 172.21.15.110:0/2817500009 reap_dead start 2024-05-04T17:39:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 shutdown_connections 2024-05-04T17:39:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 >> 172.21.15.110:0/2817500009 conn(0x7f71740b0e30 msgr2=0x7f7174070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 shutdown_connections 2024-05-04T17:39:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.046+0000 7f7178c4e640 1 -- 172.21.15.110:0/2817500009 wait complete. 2024-05-04T17:39:59.061 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-04T17:39:59.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.177+0000 7f99056c0640 1 Processor -- start 2024-05-04T17:39:59.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.177+0000 7f99056c0640 1 -- start start 2024-05-04T17:39:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.177+0000 7f99056c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9900146420 con 0x7f990013f8d0 2024-05-04T17:39:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.177+0000 7f99056c0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9900072fe0 con 0x7f9900146aa0 2024-05-04T17:39:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.177+0000 7f99056c0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99000732c0 con 0x7f9900146040 2024-05-04T17:39:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98ff7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f990013f8d0 0x7f990014e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51232/0 (socket says 172.21.15.110:51232) 2024-05-04T17:39:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98ff7fe640 1 -- 172.21.15.110:0/1892599101 learned_addr learned my addr 172.21.15.110:0/1892599101 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98feffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f9900146040 0x7f990006f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41164/0 (socket says 172.21.15.110:41164) 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3242708017 0 0) 0x7f99000732c0 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98ec003520 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795178702 0 0) 0x7f9900146420 con 0x7f990013f8d0 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99000732c0 con 0x7f990013f8d0 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647518467 0 0) 0x7f9900072fe0 con 0x7f9900146aa0 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9900146420 con 0x7f9900146aa0 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3264484567 0 0) 0x7f98ec003520 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 >> v1:172.21.15.136:6789/0 conn(0x7f9900146aa0 legacy=0x7f9900071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 >> v1:172.21.15.110:6789/0 conn(0x7f990013f8d0 legacy=0x7f990014e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9900071280 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f98f0001610 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98f0001930 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.178+0000 7f98fe7fc640 1 -- 172.21.15.110:0/1892599101 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f98f0001d00 con 0x7f9900146040 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.179+0000 7f99056c0640 1 -- 172.21.15.110:0/1892599101 >> v1:172.21.15.110:6790/0 conn(0x7f9900146040 legacy=0x7f990006f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.179+0000 7f99056c0640 1 -- 172.21.15.110:0/1892599101 shutdown_connections 2024-05-04T17:39:59.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.179+0000 7f99056c0640 1 -- 172.21.15.110:0/1892599101 >> 172.21.15.110:0/1892599101 conn(0x7f99000b5030 msgr2=0x7f99000b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:59.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.179+0000 7f99056c0640 1 -- 172.21.15.110:0/1892599101 shutdown_connections 2024-05-04T17:39:59.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.179+0000 7f99056c0640 1 -- 172.21.15.110:0/1892599101 wait complete. 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f99056c0640 1 Processor -- start 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f99056c0640 1 -- start start 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f99056c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9900141e10 con 0x7f9900146040 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f99056c0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99000ada30 con 0x7f9900146aa0 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f99056c0640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9900141ae0 con 0x7f990013f8d0 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f98feffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f990013f8d0 0x7f9900137870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41172/0 (socket says 172.21.15.110:41172) 2024-05-04T17:39:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f98feffd640 1 -- 172.21.15.110:0/1544274006 learned_addr learned my addr 172.21.15.110:0/1544274006 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.180+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774077881 0 0) 0x7f9900141ae0 con 0x7f990013f8d0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e0003520 con 0x7f990013f8d0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870457842 0 0) 0x7f9900141e10 con 0x7f9900146040 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9900141ae0 con 0x7f9900146040 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427601489 0 0) 0x7f99000ada30 con 0x7f9900146aa0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9900141e10 con 0x7f9900146aa0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2972042557 0 0) 0x7f98e0003520 con 0x7f990013f8d0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99000ada30 con 0x7f990013f8d0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f98f00015d0 con 0x7f990013f8d0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1917606510 0 0) 0x7f9900141e10 con 0x7f9900146aa0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98e0003520 con 0x7f9900146aa0 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2092758447 0 0) 0x7f9900141ae0 con 0x7f9900146040 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9900141e10 con 0x7f9900146040 2024-05-04T17:39:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f98f8001bf0 con 0x7f9900146aa0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f98e8001810 con 0x7f9900146040 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2645671006 0 0) 0x7f99000ada30 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 >> v1:172.21.15.136:6789/0 conn(0x7f9900146aa0 legacy=0x7f99001d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 >> v1:172.21.15.110:6789/0 conn(0x7f9900146040 legacy=0x7f99001d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99001d87b0 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9900137f80 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9900139770 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98f0001930 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.181+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f98f0004da0 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.182+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e4005080 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.182+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f98f0004da0 con 0x7f990013f8d0 2024-05-04T17:39:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.183+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f98f0086250 con 0x7f990013f8d0 2024-05-04T17:39:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.187+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f98f0055070 con 0x7f990013f8d0 2024-05-04T17:39:59.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.369+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f98e4000b90 con 0x7f98e006ca10 2024-05-04T17:39:59.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.370+0000 7f98f77fe640 1 -- 172.21.15.110:0/1544274006 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (unknown 1239724482 0 0) 0x7f98e4000b90 con 0x7f98e006ca10 2024-05-04T17:39:59.371 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 7 to deep-scrub 2024-05-04T17:39:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.372+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 >> v1:172.21.15.110:6816/783516658 conn(0x7f98e006ca10 legacy=0x7f98e006eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.372+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 >> v1:172.21.15.110:6790/0 conn(0x7f990013f8d0 legacy=0x7f9900137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.372+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 shutdown_connections 2024-05-04T17:39:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.372+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 >> 172.21.15.110:0/1544274006 conn(0x7f99000b5030 msgr2=0x7f990013e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.372+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 shutdown_connections 2024-05-04T17:39:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.372+0000 7f99056c0640 1 -- 172.21.15.110:0/1544274006 wait complete. 2024-05-04T17:39:59.386 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:39:59.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.500+0000 7f0a20274640 1 Processor -- start 2024-05-04T17:39:59.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.500+0000 7f0a20274640 1 -- start start 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a20274640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a18146460 con 0x7f0a18142340 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a20274640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a18073120 con 0x7f0a18146080 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a20274640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a18071400 con 0x7f0a18146710 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d80f640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0a18146710 0x7f0a18141c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41180/0 (socket says 172.21.15.110:41180) 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d80f640 1 -- 172.21.15.110:0/2528655145 learned_addr learned my addr 172.21.15.110:0/2528655145 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1e811640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0a18142340 0x7f0a1814e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51244/0 (socket says 172.21.15.110:51244) 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585187040 0 0) 0x7f0a18071400 con 0x7f0a18146710 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a04003520 con 0x7f0a18146710 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 772602712 0 0) 0x7f0a18146460 con 0x7f0a18142340 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a18071400 con 0x7f0a18142340 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.501+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395467004 0 0) 0x7f0a18073120 con 0x7f0a18146080 2024-05-04T17:39:59.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a18146460 con 0x7f0a18146080 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2169698028 0 0) 0x7f0a04003520 con 0x7f0a18146710 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 >> v1:172.21.15.136:6789/0 conn(0x7f0a18146080 legacy=0x7f0a1806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 >> v1:172.21.15.110:6789/0 conn(0x7f0a18142340 legacy=0x7f0a1814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a18071f60 con 0x7f0a18146710 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0a08001610 con 0x7f0a18146710 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a08001840 con 0x7f0a18146710 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a1d00e640 1 -- 172.21.15.110:0/2528655145 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0a08001c50 con 0x7f0a18146710 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.502+0000 7f0a20274640 1 -- 172.21.15.110:0/2528655145 >> v1:172.21.15.110:6790/0 conn(0x7f0a18146710 legacy=0x7f0a18141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.503+0000 7f0a20274640 1 -- 172.21.15.110:0/2528655145 shutdown_connections 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.503+0000 7f0a20274640 1 -- 172.21.15.110:0/2528655145 >> 172.21.15.110:0/2528655145 conn(0x7f0a180b8920 msgr2=0x7f0a180b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.503+0000 7f0a20274640 1 -- 172.21.15.110:0/2528655145 shutdown_connections 2024-05-04T17:39:59.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.503+0000 7f0a20274640 1 -- 172.21.15.110:0/2528655145 wait complete. 2024-05-04T17:39:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.504+0000 7f0a20274640 1 Processor -- start 2024-05-04T17:39:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.504+0000 7f0a20274640 1 -- start start 2024-05-04T17:39:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a20274640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a180b3ea0 con 0x7f0a18146080 2024-05-04T17:39:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a20274640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a180b3020 con 0x7f0a18142340 2024-05-04T17:39:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a20274640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a180b3300 con 0x7f0a18146710 2024-05-04T17:39:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a1d80f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0a18146080 0x7f0a180b3790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51250/0 (socket says 172.21.15.110:51250) 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a1d80f640 1 -- 172.21.15.110:0/3865991 learned_addr learned my addr 172.21.15.110:0/3865991 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919802310 0 0) 0x7f0a180b3ea0 con 0x7f0a18146080 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09fc003520 con 0x7f0a18146080 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 534068748 0 0) 0x7f0a180b3300 con 0x7f0a18146710 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a180b3ea0 con 0x7f0a18146710 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369605889 0 0) 0x7f0a180b3020 con 0x7f0a18142340 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a180b3300 con 0x7f0a18142340 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2784045560 0 0) 0x7f0a180b3ea0 con 0x7f0a18146710 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a180b3020 con 0x7f0a18146710 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0a100014b0 con 0x7f0a18146710 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2052173096 0 0) 0x7f09fc003520 con 0x7f0a18146080 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a180b3ea0 con 0x7f0a18146080 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.505+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3446343438 0 0) 0x7f0a180b3300 con 0x7f0a18142340 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09fc0054e0 con 0x7f0a18142340 2024-05-04T17:39:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 809449310 0 0) 0x7f0a180b3020 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 >> v1:172.21.15.136:6789/0 conn(0x7f0a18142340 legacy=0x7f0a181d8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 >> v1:172.21.15.110:6789/0 conn(0x7f0a18146080 legacy=0x7f0a180b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a181d7c10 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a180b1090 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a10004b60 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f0a10004eb0 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.506+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a181d79a0 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.507+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a18074a80 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.507+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f0a1001b020 con 0x7f0a18146710 2024-05-04T17:39:59.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.508+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f0a10086450 con 0x7f0a18146710 2024-05-04T17:39:59.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.515+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f0a10055130 con 0x7f0a18146710 2024-05-04T17:39:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.685+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a18062e90 con 0x7f09fc06cc80 2024-05-04T17:39:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.688+0000 7f0a0effd640 1 -- 172.21.15.110:0/3865991 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116772 (unknown 2965378022 0 3773696687) 0x7f0a18062e90 con 0x7f09fc06cc80 2024-05-04T17:39:59.691 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:39:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:39:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.690+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 >> v1:172.21.15.110:6816/783516658 conn(0x7f09fc06cc80 legacy=0x7f09fc06f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.690+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 >> v1:172.21.15.110:6790/0 conn(0x7f0a18146710 legacy=0x7f0a180b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:39:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.690+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 shutdown_connections 2024-05-04T17:39:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.690+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 >> 172.21.15.110:0/3865991 conn(0x7f0a180b8920 msgr2=0x7f0a1806bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:39:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.691+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 shutdown_connections 2024-05-04T17:39:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:39:59.691+0000 7f0a20274640 1 -- 172.21.15.110:0/3865991 wait complete. 2024-05-04T17:39:59.703 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-05-04T17:39:59.174606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3579126552,"num_objects":882,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":2686,"num_read_kb":2686,"num_write":163790,"num_write_kb":5654617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166645,"ondisk_log_size":166645,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8442880,"kb_used_data":6701536,"kb_used_omap":12,"kb_used_meta":1741235,"kb_avail":746531840,"statfs":{"total":773094113280,"available":764448604160,"internally_reserved":0,"allocated":6862372864,"data_stored":6856926288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1783025201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8720,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":66,"num_read_kb":71,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001998"},"pg_stats":[{"pgid":"3.1f","version":"21'4742","reported_seq":4778,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039418+0000","last_change":"2024-05-04T17:36:05.257969+0000","last_active":"2024-05-04T17:39:53.039418+0000","last_peered":"2024-05-04T17:39:53.039418+0000","last_clean":"2024-05-04T17:39:53.039418+0000","last_became_active":"2024-05-04T17:36:05.257710+0000","last_became_peered":"2024-05-04T17:36:05.257710+0000","last_unstale":"2024-05-04T17:39:53.039418+0000","last_undegraded":"2024-05-04T17:39:53.039418+0000","last_fullsized":"2024-05-04T17:39:53.039418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":4760,"num_write_kb":219976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"23'6419","reported_seq":6444,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.038678+0000","last_change":"2024-05-04T17:36:05.257156+0000","last_active":"2024-05-04T17:39:53.038678+0000","last_peered":"2024-05-04T17:39:53.038678+0000","last_clean":"2024-05-04T17:39:53.038678+0000","last_became_active":"2024-05-04T17:36:05.256896+0000","last_became_peered":"2024-05-04T17:36:05.256896+0000","last_unstale":"2024-05-04T17:39:53.038678+0000","last_undegraded":"2024-05-04T17:39:53.038678+0000","last_fullsized":"2024-05-04T17:39:53.038678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:10:08.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377808,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":6410,"num_write_kb":223874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"23'5314","reported_seq":5341,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039671+0000","last_change":"2024-05-04T17:36:05.260349+0000","last_active":"2024-05-04T17:39:53.039671+0000","last_peered":"2024-05-04T17:39:53.039671+0000","last_clean":"2024-05-04T17:39:53.039671+0000","last_became_active":"2024-05-04T17:36:05.260058+0000","last_became_peered":"2024-05-04T17:36:05.260058+0000","last_unstale":"2024-05-04T17:39:53.039671+0000","last_undegraded":"2024-05-04T17:39:53.039671+0000","last_fullsized":"2024-05-04T17:39:53.039671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:32:54.078676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121635328,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":5308,"num_write_kb":156688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"21'3541","reported_seq":3560,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039321+0000","last_change":"2024-05-04T17:36:05.260218+0000","last_active":"2024-05-04T17:39:53.039321+0000","last_peered":"2024-05-04T17:39:53.039321+0000","last_clean":"2024-05-04T17:39:53.039321+0000","last_became_active":"2024-05-04T17:36:05.259933+0000","last_became_peered":"2024-05-04T17:36:05.259933+0000","last_unstale":"2024-05-04T17:39:53.039321+0000","last_undegraded":"2024-05-04T17:39:53.039321+0000","last_fullsized":"2024-05-04T17:39:53.039321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:56:35.210162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":168834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"23'3390","reported_seq":3410,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.081635+0000","last_change":"2024-05-04T17:36:05.260201+0000","last_active":"2024-05-04T17:39:53.081635+0000","last_peered":"2024-05-04T17:39:53.081635+0000","last_clean":"2024-05-04T17:39:53.081635+0000","last_became_active":"2024-05-04T17:36:05.259956+0000","last_became_peered":"2024-05-04T17:36:05.259956+0000","last_unstale":"2024-05-04T17:39:53.081635+0000","last_undegraded":"2024-05-04T17:39:53.081635+0000","last_fullsized":"2024-05-04T17:39:53.081635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":156550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5221","reported_seq":5249,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.078622+0000","last_change":"2024-05-04T17:36:05.259945+0000","last_active":"2024-05-04T17:39:53.078622+0000","last_peered":"2024-05-04T17:39:53.078622+0000","last_clean":"2024-05-04T17:39:53.078622+0000","last_became_active":"2024-05-04T17:36:05.259594+0000","last_became_peered":"2024-05-04T17:36:05.259594+0000","last_unstale":"2024-05-04T17:39:53.078622+0000","last_undegraded":"2024-05-04T17:39:53.078622+0000","last_fullsized":"2024-05-04T17:39:53.078622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731664,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":28,"num_read_kb":21,"num_write":5212,"num_write_kb":167052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"23'4734","reported_seq":4754,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.080769+0000","last_change":"2024-05-04T17:36:05.250121+0000","last_active":"2024-05-04T17:39:53.080769+0000","last_peered":"2024-05-04T17:39:53.080769+0000","last_clean":"2024-05-04T17:39:53.080769+0000","last_became_active":"2024-05-04T17:36:05.249833+0000","last_became_peered":"2024-05-04T17:36:05.249833+0000","last_unstale":"2024-05-04T17:39:53.080769+0000","last_undegraded":"2024-05-04T17:39:53.080769+0000","last_fullsized":"2024-05-04T17:39:53.080769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4685,"num_write_kb":173061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"22'6599","reported_seq":6618,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.038285+0000","last_change":"2024-05-04T17:36:05.250305+0000","last_active":"2024-05-04T17:39:53.038285+0000","last_peered":"2024-05-04T17:39:53.038285+0000","last_clean":"2024-05-04T17:39:53.038285+0000","last_became_active":"2024-05-04T17:36:05.250054+0000","last_became_peered":"2024-05-04T17:36:05.250054+0000","last_unstale":"2024-05-04T17:39:53.038285+0000","last_undegraded":"2024-05-04T17:39:53.038285+0000","last_fullsized":"2024-05-04T17:39:53.038285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:20:34.169660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":157654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"22'7277","reported_seq":7300,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039445+0000","last_change":"2024-05-04T17:36:05.258270+0000","last_active":"2024-05-04T17:39:53.039445+0000","last_peered":"2024-05-04T17:39:53.039445+0000","last_clean":"2024-05-04T17:39:53.039445+0000","last_became_active":"2024-05-04T17:36:05.257748+0000","last_became_peered":"2024-05-04T17:36:05.257748+0000","last_unstale":"2024-05-04T17:39:53.039445+0000","last_undegraded":"2024-05-04T17:39:53.039445+0000","last_fullsized":"2024-05-04T17:39:53.039445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":7280,"num_write_kb":228459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"23'5970","reported_seq":6299,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039440+0000","last_change":"2024-05-04T17:36:05.264020+0000","last_active":"2024-05-04T17:39:53.039440+0000","last_peered":"2024-05-04T17:39:53.039440+0000","last_clean":"2024-05-04T17:39:53.039440+0000","last_became_active":"2024-05-04T17:36:05.263680+0000","last_became_peered":"2024-05-04T17:36:05.263680+0000","last_unstale":"2024-05-04T17:39:53.039440+0000","last_undegraded":"2024-05-04T17:39:53.039440+0000","last_fullsized":"2024-05-04T17:39:53.039440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:22:37.444606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126026248,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":1038,"num_read_kb":1054,"num_write":6173,"num_write_kb":205402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"22'8521","reported_seq":8556,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049377+0000","last_change":"2024-05-04T17:36:05.260924+0000","last_active":"2024-05-04T17:39:53.049377+0000","last_peered":"2024-05-04T17:39:53.049377+0000","last_clean":"2024-05-04T17:39:53.049377+0000","last_became_active":"2024-05-04T17:36:05.260588+0000","last_became_peered":"2024-05-04T17:36:05.260588+0000","last_unstale":"2024-05-04T17:39:53.049377+0000","last_undegraded":"2024-05-04T17:39:53.049377+0000","last_fullsized":"2024-05-04T17:39:53.049377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:20:08.528563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":50,"num_read_kb":41,"num_write":8540,"num_write_kb":185557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"23'5982","reported_seq":6018,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.085531+0000","last_change":"2024-05-04T17:36:05.262544+0000","last_active":"2024-05-04T17:39:53.085531+0000","last_peered":"2024-05-04T17:39:53.085531+0000","last_clean":"2024-05-04T17:39:53.085531+0000","last_became_active":"2024-05-04T17:36:05.262061+0000","last_became_peered":"2024-05-04T17:36:05.262061+0000","last_unstale":"2024-05-04T17:39:53.085531+0000","last_undegraded":"2024-05-04T17:39:53.085531+0000","last_fullsized":"2024-05-04T17:39:53.085531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":5961,"num_write_kb":189699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.056441+0000","last_change":"2024-05-04T17:35:59.221521+0000","last_active":"2024-05-04T17:39:53.056441+0000","last_peered":"2024-05-04T17:39:53.056441+0000","last_clean":"2024-05-04T17:39:53.056441+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:39:53.056441+0000","last_undegraded":"2024-05-04T17:39:53.056441+0000","last_fullsized":"2024-05-04T17:39:53.056441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:11:32.394654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'6197","reported_seq":6229,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:56.915187+0000","last_change":"2024-05-04T17:39:56.915259+0000","last_active":"2024-05-04T17:39:56.915187+0000","last_peered":"2024-05-04T17:39:56.915187+0000","last_clean":"2024-05-04T17:39:56.915187+0000","last_became_active":"2024-05-04T17:36:05.247697+0000","last_became_peered":"2024-05-04T17:36:05.247697+0000","last_unstale":"2024-05-04T17:39:56.915187+0000","last_undegraded":"2024-05-04T17:39:56.915187+0000","last_fullsized":"2024-05-04T17:39:56.915187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6197","last_scrub_stamp":"2024-05-04T17:39:56.915112+0000","last_deep_scrub":"23'6197","last_deep_scrub_stamp":"2024-05-04T17:39:56.915112+0000","last_clean_scrub_stamp":"2024-05-04T17:39:56.915112+0000","objects_scrubbed":21,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:06:43.614325+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78672920,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":6162,"num_write_kb":192039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.056404+0000","last_change":"2024-05-04T17:35:59.221461+0000","last_active":"2024-05-04T17:39:53.056404+0000","last_peered":"2024-05-04T17:39:53.056404+0000","last_clean":"2024-05-04T17:39:53.056404+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:39:53.056404+0000","last_undegraded":"2024-05-04T17:39:53.056404+0000","last_fullsized":"2024-05-04T17:39:53.056404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'3986","reported_seq":4012,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.080452+0000","last_change":"2024-05-04T17:36:05.262127+0000","last_active":"2024-05-04T17:39:53.080452+0000","last_peered":"2024-05-04T17:39:53.080452+0000","last_clean":"2024-05-04T17:39:53.080452+0000","last_became_active":"2024-05-04T17:36:05.261828+0000","last_became_peered":"2024-05-04T17:36:05.261828+0000","last_unstale":"2024-05-04T17:39:53.080452+0000","last_undegraded":"2024-05-04T17:39:53.080452+0000","last_fullsized":"2024-05-04T17:39:53.080452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:57:34.308542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":3961,"num_write_kb":185542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049797+0000","last_change":"2024-05-04T17:35:59.222186+0000","last_active":"2024-05-04T17:39:53.049797+0000","last_peered":"2024-05-04T17:39:53.049797+0000","last_clean":"2024-05-04T17:39:53.049797+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:39:53.049797+0000","last_undegraded":"2024-05-04T17:39:53.049797+0000","last_fullsized":"2024-05-04T17:39:53.049797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:15:38.191128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'5132","reported_seq":5152,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.056337+0000","last_change":"2024-05-04T17:36:05.395570+0000","last_active":"2024-05-04T17:39:53.056337+0000","last_peered":"2024-05-04T17:39:53.056337+0000","last_clean":"2024-05-04T17:39:53.056337+0000","last_became_active":"2024-05-04T17:36:05.395267+0000","last_became_peered":"2024-05-04T17:36:05.395267+0000","last_unstale":"2024-05-04T17:39:53.056337+0000","last_undegraded":"2024-05-04T17:39:53.056337+0000","last_fullsized":"2024-05-04T17:39:53.056337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":103041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.056392+0000","last_change":"2024-05-04T17:35:59.218744+0000","last_active":"2024-05-04T17:39:53.056392+0000","last_peered":"2024-05-04T17:39:53.056392+0000","last_clean":"2024-05-04T17:39:53.056392+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:39:53.056392+0000","last_undegraded":"2024-05-04T17:39:53.056392+0000","last_fullsized":"2024-05-04T17:39:53.056392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'6471","reported_seq":6521,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039714+0000","last_change":"2024-05-04T17:36:05.254630+0000","last_active":"2024-05-04T17:39:53.039714+0000","last_peered":"2024-05-04T17:39:53.039714+0000","last_clean":"2024-05-04T17:39:53.039714+0000","last_became_active":"2024-05-04T17:36:05.254049+0000","last_became_peered":"2024-05-04T17:36:05.254049+0000","last_unstale":"2024-05-04T17:39:53.039714+0000","last_undegraded":"2024-05-04T17:39:53.039714+0000","last_fullsized":"2024-05-04T17:39:53.039714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:59:05.699546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123508560,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":117,"num_read_kb":157,"num_write":6516,"num_write_kb":216324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'4380","reported_seq":4405,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.041829+0000","last_change":"2024-05-04T17:36:05.260042+0000","last_active":"2024-05-04T17:39:53.041829+0000","last_peered":"2024-05-04T17:39:53.041829+0000","last_clean":"2024-05-04T17:39:53.041829+0000","last_became_active":"2024-05-04T17:36:05.259684+0000","last_became_peered":"2024-05-04T17:36:05.259684+0000","last_unstale":"2024-05-04T17:39:53.041829+0000","last_undegraded":"2024-05-04T17:39:53.041829+0000","last_fullsized":"2024-05-04T17:39:53.041829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829632,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":4394,"num_write_kb":197895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039362+0000","last_change":"2024-05-04T17:35:59.223094+0000","last_active":"2024-05-04T17:39:53.039362+0000","last_peered":"2024-05-04T17:39:53.039362+0000","last_clean":"2024-05-04T17:39:53.039362+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:39:53.039362+0000","last_undegraded":"2024-05-04T17:39:53.039362+0000","last_fullsized":"2024-05-04T17:39:53.039362+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:54:23.823020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"23'3966","reported_seq":3986,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.080016+0000","last_change":"2024-05-04T17:36:05.395580+0000","last_active":"2024-05-04T17:39:53.080016+0000","last_peered":"2024-05-04T17:39:53.080016+0000","last_clean":"2024-05-04T17:39:53.080016+0000","last_became_active":"2024-05-04T17:36:05.395272+0000","last_became_peered":"2024-05-04T17:36:05.395272+0000","last_unstale":"2024-05-04T17:39:53.080016+0000","last_undegraded":"2024-05-04T17:39:53.080016+0000","last_fullsized":"2024-05-04T17:39:53.080016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110366368,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":158385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.045892+0000","last_change":"2024-05-04T17:35:59.331384+0000","last_active":"2024-05-04T17:39:53.045892+0000","last_peered":"2024-05-04T17:39:53.045892+0000","last_clean":"2024-05-04T17:39:53.045892+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:39:53.045892+0000","last_undegraded":"2024-05-04T17:39:53.045892+0000","last_fullsized":"2024-05-04T17:39:53.045892+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"23'4101","reported_seq":4121,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.079125+0000","last_change":"2024-05-04T17:36:05.261680+0000","last_active":"2024-05-04T17:39:53.079125+0000","last_peered":"2024-05-04T17:39:53.079125+0000","last_clean":"2024-05-04T17:39:53.079125+0000","last_became_active":"2024-05-04T17:36:05.261395+0000","last_became_peered":"2024-05-04T17:36:05.261395+0000","last_unstale":"2024-05-04T17:39:53.079125+0000","last_undegraded":"2024-05-04T17:39:53.079125+0000","last_fullsized":"2024-05-04T17:39:53.079125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106299392,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":186884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049338+0000","last_change":"2024-05-04T17:35:59.223322+0000","last_active":"2024-05-04T17:39:53.049338+0000","last_peered":"2024-05-04T17:39:53.049338+0000","last_clean":"2024-05-04T17:39:53.049338+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:39:53.049338+0000","last_undegraded":"2024-05-04T17:39:53.049338+0000","last_fullsized":"2024-05-04T17:39:53.049338+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:34:10.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"22'5762","reported_seq":5781,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.038314+0000","last_change":"2024-05-04T17:36:05.258017+0000","last_active":"2024-05-04T17:39:53.038314+0000","last_peered":"2024-05-04T17:39:53.038314+0000","last_clean":"2024-05-04T17:39:53.038314+0000","last_became_active":"2024-05-04T17:36:05.257750+0000","last_became_peered":"2024-05-04T17:36:05.257750+0000","last_unstale":"2024-05-04T17:39:53.038314+0000","last_undegraded":"2024-05-04T17:39:53.038314+0000","last_fullsized":"2024-05-04T17:39:53.038314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:20:27.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":181199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.039321+0000","last_change":"2024-05-04T17:35:59.221969+0000","last_active":"2024-05-04T17:39:53.039321+0000","last_peered":"2024-05-04T17:39:53.039321+0000","last_clean":"2024-05-04T17:39:53.039321+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:39:53.039321+0000","last_undegraded":"2024-05-04T17:39:53.039321+0000","last_fullsized":"2024-05-04T17:39:53.039321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:58.162933+0000","last_clean_scrub_stamp":"2024-05-04T17:35:58.162933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:22:19.190308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049422+0000","last_change":"2024-05-04T17:35:58.172775+0000","last_active":"2024-05-04T17:39:53.049422+0000","last_peered":"2024-05-04T17:39:53.049422+0000","last_clean":"2024-05-04T17:39:53.049422+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:39:53.049422+0000","last_undegraded":"2024-05-04T17:39:53.049422+0000","last_fullsized":"2024-05-04T17:39:53.049422+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:35:57.156954+0000","last_clean_scrub_stamp":"2024-05-04T17:35:57.156954+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:43:38.793258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"23'2733","reported_seq":2763,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.079570+0000","last_change":"2024-05-04T17:36:05.250179+0000","last_active":"2024-05-04T17:39:53.079570+0000","last_peered":"2024-05-04T17:39:53.079570+0000","last_clean":"2024-05-04T17:39:53.079570+0000","last_became_active":"2024-05-04T17:36:05.249926+0000","last_became_peered":"2024-05-04T17:36:05.249926+0000","last_unstale":"2024-05-04T17:39:53.079570+0000","last_undegraded":"2024-05-04T17:39:53.079570+0000","last_fullsized":"2024-05-04T17:39:53.079570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:51:15.283094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":2682,"num_write_kb":127745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"23'9946","reported_seq":10089,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.106504+0000","last_change":"2024-05-04T17:36:05.395659+0000","last_active":"2024-05-04T17:39:53.106504+0000","last_peered":"2024-05-04T17:39:53.106504+0000","last_clean":"2024-05-04T17:39:53.106504+0000","last_became_active":"2024-05-04T17:36:05.395416+0000","last_became_peered":"2024-05-04T17:36:05.395416+0000","last_unstale":"2024-05-04T17:39:53.106504+0000","last_undegraded":"2024-05-04T17:39:53.106504+0000","last_fullsized":"2024-05-04T17:39:53.106504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":386,"num_read_kb":352,"num_write":6860,"num_write_kb":187028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"23'4847","reported_seq":4869,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.056472+0000","last_change":"2024-05-04T17:36:05.257442+0000","last_active":"2024-05-04T17:39:53.056472+0000","last_peered":"2024-05-04T17:39:53.056472+0000","last_clean":"2024-05-04T17:39:53.056472+0000","last_became_active":"2024-05-04T17:36:05.257078+0000","last_became_peered":"2024-05-04T17:36:05.257078+0000","last_unstale":"2024-05-04T17:39:53.056472+0000","last_undegraded":"2024-05-04T17:39:53.056472+0000","last_fullsized":"2024-05-04T17:39:53.056472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:53:55.417005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4826,"num_write_kb":165126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"21'4340","reported_seq":4362,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.038351+0000","last_change":"2024-05-04T17:36:05.250165+0000","last_active":"2024-05-04T17:39:53.038351+0000","last_peered":"2024-05-04T17:39:53.038351+0000","last_clean":"2024-05-04T17:39:53.038351+0000","last_became_active":"2024-05-04T17:36:05.249845+0000","last_became_peered":"2024-05-04T17:36:05.249845+0000","last_unstale":"2024-05-04T17:39:53.038351+0000","last_undegraded":"2024-05-04T17:39:53.038351+0000","last_fullsized":"2024-05-04T17:39:53.038351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:40:29.446049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":4449,"num_write_kb":193090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"23'5662","reported_seq":5806,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.083833+0000","last_change":"2024-05-04T17:36:05.258056+0000","last_active":"2024-05-04T17:39:53.083833+0000","last_peered":"2024-05-04T17:39:53.083833+0000","last_clean":"2024-05-04T17:39:53.083833+0000","last_became_active":"2024-05-04T17:36:05.257760+0000","last_became_peered":"2024-05-04T17:36:05.257760+0000","last_unstale":"2024-05-04T17:39:53.083833+0000","last_undegraded":"2024-05-04T17:39:53.083833+0000","last_fullsized":"2024-05-04T17:39:53.083833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":5662,"log_dups_size":0,"ondisk_log_size":5662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:21:44.622756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140053176,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":196,"num_read_kb":182,"num_write":5664,"num_write_kb":217362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"23'3532","reported_seq":3568,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049498+0000","last_change":"2024-05-04T17:36:05.261546+0000","last_active":"2024-05-04T17:39:53.049498+0000","last_peered":"2024-05-04T17:39:53.049498+0000","last_clean":"2024-05-04T17:39:53.049498+0000","last_became_active":"2024-05-04T17:36:05.261259+0000","last_became_peered":"2024-05-04T17:36:05.261259+0000","last_unstale":"2024-05-04T17:39:53.049498+0000","last_undegraded":"2024-05-04T17:39:53.049498+0000","last_fullsized":"2024-05-04T17:39:53.049498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:56:45.912709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":66,"num_read_kb":115,"num_write":3541,"num_write_kb":181187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"23'4853","reported_seq":4964,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.088690+0000","last_change":"2024-05-04T17:36:05.263514+0000","last_active":"2024-05-04T17:39:53.088690+0000","last_peered":"2024-05-04T17:39:53.088690+0000","last_clean":"2024-05-04T17:39:53.088690+0000","last_became_active":"2024-05-04T17:36:05.263214+0000","last_became_peered":"2024-05-04T17:36:05.263214+0000","last_unstale":"2024-05-04T17:39:53.088690+0000","last_undegraded":"2024-05-04T17:39:53.088690+0000","last_fullsized":"2024-05-04T17:39:53.088690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:05.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":300,"num_read_kb":235,"num_write":5104,"num_write_kb":162415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"23'3519","reported_seq":3538,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.082994+0000","last_change":"2024-05-04T17:36:05.263191+0000","last_active":"2024-05-04T17:39:53.082994+0000","last_peered":"2024-05-04T17:39:53.082994+0000","last_clean":"2024-05-04T17:39:53.082994+0000","last_became_active":"2024-05-04T17:36:05.262817+0000","last_became_peered":"2024-05-04T17:36:05.262817+0000","last_unstale":"2024-05-04T17:39:53.082994+0000","last_undegraded":"2024-05-04T17:39:53.082994+0000","last_fullsized":"2024-05-04T17:39:53.082994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:17:30.844586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80883712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":134410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"23'3811","reported_seq":3832,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.081673+0000","last_change":"2024-05-04T17:36:05.250207+0000","last_active":"2024-05-04T17:39:53.081673+0000","last_peered":"2024-05-04T17:39:53.081673+0000","last_clean":"2024-05-04T17:39:53.081673+0000","last_became_active":"2024-05-04T17:36:05.249932+0000","last_became_peered":"2024-05-04T17:36:05.249932+0000","last_unstale":"2024-05-04T17:39:53.081673+0000","last_undegraded":"2024-05-04T17:39:53.081673+0000","last_fullsized":"2024-05-04T17:39:53.081673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:23:37.128989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112574992,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3768,"num_write_kb":163450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"21'3909","reported_seq":3935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049352+0000","last_change":"2024-05-04T17:36:05.263401+0000","last_active":"2024-05-04T17:39:53.049352+0000","last_peered":"2024-05-04T17:39:53.049352+0000","last_clean":"2024-05-04T17:39:53.049352+0000","last_became_active":"2024-05-04T17:36:05.263000+0000","last_became_peered":"2024-05-04T17:36:05.263000+0000","last_unstale":"2024-05-04T17:39:53.049352+0000","last_undegraded":"2024-05-04T17:39:53.049352+0000","last_fullsized":"2024-05-04T17:39:53.049352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3909,"log_dups_size":0,"ondisk_log_size":3909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:51:16.339176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":3923,"num_write_kb":150485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"23'3429","reported_seq":3462,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.078384+0000","last_change":"2024-05-04T17:36:05.261174+0000","last_active":"2024-05-04T17:39:53.078384+0000","last_peered":"2024-05-04T17:39:53.078384+0000","last_clean":"2024-05-04T17:39:53.078384+0000","last_became_active":"2024-05-04T17:36:05.260881+0000","last_became_peered":"2024-05-04T17:36:05.260881+0000","last_unstale":"2024-05-04T17:39:53.078384+0000","last_undegraded":"2024-05-04T17:39:53.078384+0000","last_fullsized":"2024-05-04T17:39:53.078384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":54,"num_read_kb":41,"num_write":3408,"num_write_kb":117896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"23'8242","reported_seq":8264,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:53.049444+0000","last_change":"2024-05-04T17:36:05.263104+0000","last_active":"2024-05-04T17:39:53.049444+0000","last_peered":"2024-05-04T17:39:53.049444+0000","last_clean":"2024-05-04T17:39:53.049444+0000","last_became_active":"2024-05-04T17:36:05.262502+0000","last_became_peered":"2024-05-04T17:36:05.262502+0000","last_unstale":"2024-05-04T17:39:53.049444+0000","last_undegraded":"2024-05-04T17:39:53.049444+0000","last_fullsized":"2024-05-04T17:39:53.049444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:36:04.237763+0000","last_clean_scrub_stamp":"2024-05-04T17:36:04.237763+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:53:50.786398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8220,"num_write_kb":197892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3578536184,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":2570,"num_read_kb":2586,"num_write":163631,"num_write_kb":5652201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6859538432,"data_stored":6855271544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166528,"ondisk_log_size":166528,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542194,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343512,"kb_used_data":1055564,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93028328,"statfs":{"total":96636764160,"available":95261007872,"internally_reserved":0,"allocated":1080897536,"data_stored":1080007567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":294844874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266632,"kb_used_data":1005052,"kb_used_omap":1,"kb_used_meta":261566,"kb_avail":93105208,"statfs":{"total":96636764160,"available":95339732992,"internally_reserved":0,"allocated":1029173248,"data_stored":1028426019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267844037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542196,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1724736,"kb_used_data":1350772,"kb_used_omap":1,"kb_used_meta":373950,"kb_avail":92647104,"statfs":{"total":96636764160,"available":94870634496,"internally_reserved":0,"allocated":1383190528,"data_stored":1382086063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":382925253},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401628,"kb_used_data":330256,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93970212,"statfs":{"total":96636764160,"available":96225497088,"internally_reserved":0,"allocated":338182144,"data_stored":337870963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362152,"kb_used_data":1074204,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93009688,"statfs":{"total":96636764160,"available":95241920512,"internally_reserved":0,"allocated":1099984896,"data_stored":1099156671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294844871},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377684,"kb_used_data":288200,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93994156,"statfs":{"total":96636764160,"available":96250015744,"internally_reserved":0,"allocated":295116800,"data_stored":294765651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649124,"kb_used_data":545816,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93722716,"statfs":{"total":96636764160,"available":95972061184,"internally_reserved":0,"allocated":558915584,"data_stored":558501959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1317412,"kb_used_data":1051672,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":93054428,"statfs":{"total":96636764160,"available":95287734272,"internally_reserved":0,"allocated":1076912128,"data_stored":1076111395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272103877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1076158464,"data_stored":1075507856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":558694400,"data_stored":558427348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294895616,"data_stored":294691040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1099763712,"data_stored":1099082060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1382969344,"data_stored":1382011452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1028952064,"data_stored":1028351408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":337960960,"data_stored":337796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080143872,"data_stored":1079404028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:39:59.709 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.710 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.711 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.712 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.712 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.712 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.712 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-04T17:35:58.162933+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=58, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-04T17:35:57.156954+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=35, tm_sec=57, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.713 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-05-04T17:36:04.237763+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=36, tm_sec=4, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=39, tm_sec=54, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T17:39:59.714 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-04T17:40:19.715 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:40:19.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.826+0000 7f6603874640 1 Processor -- start 2024-05-04T17:40:19.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.826+0000 7f6603874640 1 -- start start 2024-05-04T17:40:19.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f6603874640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc06ee70 con 0x7f65fc06ea90 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f6603874640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0a9d80 con 0x7f65fc13f5c0 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f6603874640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0aa060 con 0x7f65fc072530 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f6601610640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f65fc06ea90 0x7f65fc13eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53364/0 (socket says 172.21.15.110:53364) 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f6601610640 1 -- 172.21.15.110:0/3818586534 learned_addr learned my addr 172.21.15.110:0/3818586534 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706582251 0 0) 0x7f65fc0aa060 con 0x7f65fc072530 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e8003520 con 0x7f65fc072530 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160671895 0 0) 0x7f65fc06ee70 con 0x7f65fc06ea90 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc0aa060 con 0x7f65fc06ea90 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741548419 0 0) 0x7f65fc0a9d80 con 0x7f65fc13f5c0 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc06ee70 con 0x7f65fc13f5c0 2024-05-04T17:40:19.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2112761073 0 0) 0x7f65fc0aa060 con 0x7f65fc06ea90 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 >> v1:172.21.15.110:6790/0 conn(0x7f65fc072530 legacy=0x7f65fc143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 >> v1:172.21.15.136:6789/0 conn(0x7f65fc13f5c0 legacy=0x7f65fc14aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc142c50 con 0x7f65fc06ea90 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f65f80015f0 con 0x7f65fc06ea90 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65f8001860 con 0x7f65fc06ea90 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.827+0000 7f65f3fff640 1 -- 172.21.15.110:0/3818586534 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f65f8001c70 con 0x7f65fc06ea90 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.828+0000 7f6603874640 1 -- 172.21.15.110:0/3818586534 >> v1:172.21.15.110:6789/0 conn(0x7f65fc06ea90 legacy=0x7f65fc13eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.828+0000 7f6603874640 1 -- 172.21.15.110:0/3818586534 shutdown_connections 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.828+0000 7f6603874640 1 -- 172.21.15.110:0/3818586534 >> 172.21.15.110:0/3818586534 conn(0x7f65fc0b43a0 msgr2=0x7f65fc0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.828+0000 7f6603874640 1 -- 172.21.15.110:0/3818586534 shutdown_connections 2024-05-04T17:40:19.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.828+0000 7f6603874640 1 -- 172.21.15.110:0/3818586534 wait complete. 2024-05-04T17:40:19.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.829+0000 7f6603874640 1 Processor -- start 2024-05-04T17:40:19.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.829+0000 7f6603874640 1 -- start start 2024-05-04T17:40:19.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6603874640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0aef70 con 0x7f65fc072530 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6603874640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc138d70 con 0x7f65fc13f5c0 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6603874640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0ae8c0 con 0x7f65fc06ea90 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6601610640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f65fc06ea90 0x7f65fc1d89e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33854/0 (socket says 172.21.15.110:33854) 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6600e0f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f65fc072530 0x7f65fc1d90f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53374/0 (socket says 172.21.15.110:53374) 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6601610640 1 -- 172.21.15.110:0/3810088459 learned_addr learned my addr 172.21.15.110:0/3810088459 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767690334 0 0) 0x7f65fc0aef70 con 0x7f65fc072530 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e0003520 con 0x7f65fc072530 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006745187 0 0) 0x7f65fc0ae8c0 con 0x7f65fc06ea90 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc0aef70 con 0x7f65fc06ea90 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3137900366 0 0) 0x7f65fc138d70 con 0x7f65fc13f5c0 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc0ae8c0 con 0x7f65fc13f5c0 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3913089236 0 0) 0x7f65e0003520 con 0x7f65fc072530 2024-05-04T17:40:19.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65fc138d70 con 0x7f65fc072530 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f65ec001a70 con 0x7f65fc072530 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 664842856 0 0) 0x7f65fc0aef70 con 0x7f65fc06ea90 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65e0003520 con 0x7f65fc06ea90 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f65f8001bb0 con 0x7f65fc06ea90 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1592992627 0 0) 0x7f65fc0ae8c0 con 0x7f65fc13f5c0 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65fc0aef70 con 0x7f65fc13f5c0 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f65f4001b90 con 0x7f65fc13f5c0 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1862645932 0 0) 0x7f65fc138d70 con 0x7f65fc072530 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 >> v1:172.21.15.110:6790/0 conn(0x7f65fc06ea90 legacy=0x7f65fc1d89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 >> v1:172.21.15.136:6789/0 conn(0x7f65fc13f5c0 legacy=0x7f65fc1dff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc0aea90 con 0x7f65fc072530 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.830+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc0b2f20 con 0x7f65fc072530 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.831+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65ec001d20 con 0x7f65fc072530 2024-05-04T17:40:19.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.831+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f65ec003a40 con 0x7f65fc072530 2024-05-04T17:40:19.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.831+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65fc0add90 con 0x7f65fc072530 2024-05-04T17:40:19.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.832+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65fc062e90 con 0x7f65fc072530 2024-05-04T17:40:19.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.832+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f65ec003ca0 con 0x7f65fc072530 2024-05-04T17:40:19.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.832+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f65ec0860e0 con 0x7f65fc072530 2024-05-04T17:40:19.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:19.837+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f65ec054f00 con 0x7f65fc072530 2024-05-04T17:40:20.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.005+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 --> v1:172.21.15.110:6816/783516658 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f65fc068a40 con 0x7f65e006c970 2024-05-04T17:40:20.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.008+0000 7f65f1ffb640 1 -- 172.21.15.110:0/3810088459 <== mgr.4110 v1:172.21.15.110:6816/783516658 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117639 (unknown 2965378022 0 4058991984) 0x7f65fc068a40 con 0x7f65e006c970 2024-05-04T17:40:20.010 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-04T17:40:20.011 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-05-04T17:40:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.010+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 >> v1:172.21.15.110:6816/783516658 conn(0x7f65e006c970 legacy=0x7f65e006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.010+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 >> v1:172.21.15.110:6789/0 conn(0x7f65fc072530 legacy=0x7f65fc1d90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.010+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 shutdown_connections 2024-05-04T17:40:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.010+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 >> 172.21.15.110:0/3810088459 conn(0x7f65fc0b43a0 msgr2=0x7f65fc13e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:40:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.011+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 shutdown_connections 2024-05-04T17:40:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.011+0000 7f6603874640 1 -- 172.21.15.110:0/3810088459 wait complete. 2024-05-04T17:40:20.021 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-05-04T17:40:19.178933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3579126552,"num_objects":882,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":2686,"num_read_kb":2686,"num_write":163790,"num_write_kb":5654617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167503,"ondisk_log_size":167503,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8451168,"kb_used_data":6701536,"kb_used_omap":12,"kb_used_meta":1749555,"kb_avail":746523552,"statfs":{"total":773094113280,"available":764440117248,"internally_reserved":0,"allocated":6862372864,"data_stored":6856926288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1791544881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"3.1f","version":"23'4772","reported_seq":4818,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:58.002528+0000","last_change":"2024-05-04T17:39:58.002528+0000","last_active":"2024-05-04T17:39:58.002528+0000","last_peered":"2024-05-04T17:39:58.002528+0000","last_clean":"2024-05-04T17:39:58.002528+0000","last_became_active":"2024-05-04T17:36:05.257710+0000","last_became_peered":"2024-05-04T17:36:05.257710+0000","last_unstale":"2024-05-04T17:39:58.002528+0000","last_undegraded":"2024-05-04T17:39:58.002528+0000","last_fullsized":"2024-05-04T17:39:58.002528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4772","last_scrub_stamp":"2024-05-04T17:39:58.002427+0000","last_deep_scrub":"23'4772","last_deep_scrub_stamp":"2024-05-04T17:39:58.002427+0000","last_clean_scrub_stamp":"2024-05-04T17:39:58.002427+0000","objects_scrubbed":30,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:11:08.241820+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":4760,"num_write_kb":219976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"23'6447","reported_seq":6483,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.257470+0000","last_change":"2024-05-04T17:40:01.257470+0000","last_active":"2024-05-04T17:40:01.257470+0000","last_peered":"2024-05-04T17:40:01.257470+0000","last_clean":"2024-05-04T17:40:01.257470+0000","last_became_active":"2024-05-04T17:36:05.256896+0000","last_became_peered":"2024-05-04T17:36:05.256896+0000","last_unstale":"2024-05-04T17:40:01.257470+0000","last_undegraded":"2024-05-04T17:40:01.257470+0000","last_fullsized":"2024-05-04T17:40:01.257470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6447","last_scrub_stamp":"2024-05-04T17:40:01.257372+0000","last_deep_scrub":"23'6447","last_deep_scrub_stamp":"2024-05-04T17:40:01.257372+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.257372+0000","objects_scrubbed":28,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:44:07.647920+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377808,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":6410,"num_write_kb":223874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"23'5345","reported_seq":5383,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:04.572652+0000","last_change":"2024-05-04T17:40:04.572690+0000","last_active":"2024-05-04T17:40:04.572652+0000","last_peered":"2024-05-04T17:40:04.572652+0000","last_clean":"2024-05-04T17:40:04.572652+0000","last_became_active":"2024-05-04T17:36:05.260058+0000","last_became_peered":"2024-05-04T17:36:05.260058+0000","last_unstale":"2024-05-04T17:40:04.572652+0000","last_undegraded":"2024-05-04T17:40:04.572652+0000","last_fullsized":"2024-05-04T17:40:04.572652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5345","last_scrub_stamp":"2024-05-04T17:40:04.572577+0000","last_deep_scrub":"23'5345","last_deep_scrub_stamp":"2024-05-04T17:40:04.572577+0000","last_clean_scrub_stamp":"2024-05-04T17:40:04.572577+0000","objects_scrubbed":31,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:50:26.252116+0000","scrub_duration":260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121635328,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":5308,"num_write_kb":156688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"23'3561","reported_seq":3589,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:03.465274+0000","last_change":"2024-05-04T17:40:03.465324+0000","last_active":"2024-05-04T17:40:03.465274+0000","last_peered":"2024-05-04T17:40:03.465274+0000","last_clean":"2024-05-04T17:40:03.465274+0000","last_became_active":"2024-05-04T17:36:05.259933+0000","last_became_peered":"2024-05-04T17:36:05.259933+0000","last_unstale":"2024-05-04T17:40:03.465274+0000","last_undegraded":"2024-05-04T17:40:03.465274+0000","last_fullsized":"2024-05-04T17:40:03.465274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3561","last_scrub_stamp":"2024-05-04T17:40:03.465208+0000","last_deep_scrub":"23'3561","last_deep_scrub_stamp":"2024-05-04T17:40:03.465208+0000","last_clean_scrub_stamp":"2024-05-04T17:40:03.465208+0000","objects_scrubbed":20,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:42:28.563023+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":168834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"23'3415","reported_seq":3444,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.985760+0000","last_change":"2024-05-04T17:39:59.985800+0000","last_active":"2024-05-04T17:39:59.985760+0000","last_peered":"2024-05-04T17:39:59.985760+0000","last_clean":"2024-05-04T17:39:59.985760+0000","last_became_active":"2024-05-04T17:36:05.259956+0000","last_became_peered":"2024-05-04T17:36:05.259956+0000","last_unstale":"2024-05-04T17:39:59.985760+0000","last_undegraded":"2024-05-04T17:39:59.985760+0000","last_fullsized":"2024-05-04T17:39:59.985760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3415","last_scrub_stamp":"2024-05-04T17:39:59.985638+0000","last_deep_scrub":"23'3415","last_deep_scrub_stamp":"2024-05-04T17:39:59.985638+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.985638+0000","objects_scrubbed":25,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:38:21.163451+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":156550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5246","reported_seq":5283,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:58.836735+0000","last_change":"2024-05-04T17:39:58.836765+0000","last_active":"2024-05-04T17:39:58.836735+0000","last_peered":"2024-05-04T17:39:58.836735+0000","last_clean":"2024-05-04T17:39:58.836735+0000","last_became_active":"2024-05-04T17:36:05.259594+0000","last_became_peered":"2024-05-04T17:36:05.259594+0000","last_unstale":"2024-05-04T17:39:58.836735+0000","last_undegraded":"2024-05-04T17:39:58.836735+0000","last_fullsized":"2024-05-04T17:39:58.836735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5246","last_scrub_stamp":"2024-05-04T17:39:58.836675+0000","last_deep_scrub":"23'5246","last_deep_scrub_stamp":"2024-05-04T17:39:58.836675+0000","last_clean_scrub_stamp":"2024-05-04T17:39:58.836675+0000","objects_scrubbed":25,"log_size":5246,"log_dups_size":0,"ondisk_log_size":5246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:42:03.553932+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731664,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":28,"num_read_kb":21,"num_write":5212,"num_write_kb":167052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"23'4764","reported_seq":4793,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.520741+0000","last_change":"2024-05-04T17:39:59.520775+0000","last_active":"2024-05-04T17:39:59.520741+0000","last_peered":"2024-05-04T17:39:59.520741+0000","last_clean":"2024-05-04T17:39:59.520741+0000","last_became_active":"2024-05-04T17:36:05.249833+0000","last_became_peered":"2024-05-04T17:36:05.249833+0000","last_unstale":"2024-05-04T17:39:59.520741+0000","last_undegraded":"2024-05-04T17:39:59.520741+0000","last_fullsized":"2024-05-04T17:39:59.520741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4764","last_scrub_stamp":"2024-05-04T17:39:59.520678+0000","last_deep_scrub":"23'4764","last_deep_scrub_stamp":"2024-05-04T17:39:59.520678+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.520678+0000","objects_scrubbed":30,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:09:37.417223+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4685,"num_write_kb":173061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"23'6622","reported_seq":6650,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.202588+0000","last_change":"2024-05-04T17:40:00.202678+0000","last_active":"2024-05-04T17:40:00.202588+0000","last_peered":"2024-05-04T17:40:00.202588+0000","last_clean":"2024-05-04T17:40:00.202588+0000","last_became_active":"2024-05-04T17:36:05.250054+0000","last_became_peered":"2024-05-04T17:36:05.250054+0000","last_unstale":"2024-05-04T17:40:00.202588+0000","last_undegraded":"2024-05-04T17:40:00.202588+0000","last_fullsized":"2024-05-04T17:40:00.202588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6622","last_scrub_stamp":"2024-05-04T17:40:00.202512+0000","last_deep_scrub":"23'6622","last_deep_scrub_stamp":"2024-05-04T17:40:00.202512+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.202512+0000","objects_scrubbed":23,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:18:35.725022+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":157654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"23'7308","reported_seq":7340,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.021492+0000","last_change":"2024-05-04T17:39:59.021531+0000","last_active":"2024-05-04T17:39:59.021492+0000","last_peered":"2024-05-04T17:39:59.021492+0000","last_clean":"2024-05-04T17:39:59.021492+0000","last_became_active":"2024-05-04T17:36:05.257748+0000","last_became_peered":"2024-05-04T17:36:05.257748+0000","last_unstale":"2024-05-04T17:39:59.021492+0000","last_undegraded":"2024-05-04T17:39:59.021492+0000","last_fullsized":"2024-05-04T17:39:59.021492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7308","last_scrub_stamp":"2024-05-04T17:39:59.021431+0000","last_deep_scrub":"23'7308","last_deep_scrub_stamp":"2024-05-04T17:39:59.021431+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.021431+0000","objects_scrubbed":31,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:53:57.558818+0000","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":7280,"num_write_kb":228459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"23'6002","reported_seq":6341,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:05.603293+0000","last_change":"2024-05-04T17:40:05.603332+0000","last_active":"2024-05-04T17:40:05.603293+0000","last_peered":"2024-05-04T17:40:05.603293+0000","last_clean":"2024-05-04T17:40:05.603293+0000","last_became_active":"2024-05-04T17:36:05.263680+0000","last_became_peered":"2024-05-04T17:36:05.263680+0000","last_unstale":"2024-05-04T17:40:05.603293+0000","last_undegraded":"2024-05-04T17:40:05.603293+0000","last_fullsized":"2024-05-04T17:40:05.603293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6002","last_scrub_stamp":"2024-05-04T17:40:05.603226+0000","last_deep_scrub":"23'6002","last_deep_scrub_stamp":"2024-05-04T17:40:05.603226+0000","last_clean_scrub_stamp":"2024-05-04T17:40:05.603226+0000","objects_scrubbed":32,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:56:37.239750+0000","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126026248,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":1038,"num_read_kb":1054,"num_write":6173,"num_write_kb":205402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"23'8552","reported_seq":8597,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:03.061248+0000","last_change":"2024-05-04T17:40:03.061309+0000","last_active":"2024-05-04T17:40:03.061248+0000","last_peered":"2024-05-04T17:40:03.061248+0000","last_clean":"2024-05-04T17:40:03.061248+0000","last_became_active":"2024-05-04T17:36:05.260588+0000","last_became_peered":"2024-05-04T17:36:05.260588+0000","last_unstale":"2024-05-04T17:40:03.061248+0000","last_undegraded":"2024-05-04T17:40:03.061248+0000","last_fullsized":"2024-05-04T17:40:03.061248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8552","last_scrub_stamp":"2024-05-04T17:40:03.061130+0000","last_deep_scrub":"23'8552","last_deep_scrub_stamp":"2024-05-04T17:40:03.061130+0000","last_clean_scrub_stamp":"2024-05-04T17:40:03.061130+0000","objects_scrubbed":31,"log_size":8552,"log_dups_size":0,"ondisk_log_size":8552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:52:13.088027+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":50,"num_read_kb":41,"num_write":8540,"num_write_kb":185557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"23'6002","reported_seq":6046,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.855533+0000","last_change":"2024-05-04T17:40:00.855609+0000","last_active":"2024-05-04T17:40:00.855533+0000","last_peered":"2024-05-04T17:40:00.855533+0000","last_clean":"2024-05-04T17:40:00.855533+0000","last_became_active":"2024-05-04T17:36:05.262061+0000","last_became_peered":"2024-05-04T17:36:05.262061+0000","last_unstale":"2024-05-04T17:40:00.855533+0000","last_undegraded":"2024-05-04T17:40:00.855533+0000","last_fullsized":"2024-05-04T17:40:00.855533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6002","last_scrub_stamp":"2024-05-04T17:40:00.855465+0000","last_deep_scrub":"23'6002","last_deep_scrub_stamp":"2024-05-04T17:40:00.855465+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.855465+0000","objects_scrubbed":20,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:12:06.534247+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":5961,"num_write_kb":189699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.459303+0000","last_change":"2024-05-04T17:39:59.459324+0000","last_active":"2024-05-04T17:39:59.459303+0000","last_peered":"2024-05-04T17:39:59.459303+0000","last_clean":"2024-05-04T17:39:59.459303+0000","last_became_active":"2024-05-04T17:35:59.221225+0000","last_became_peered":"2024-05-04T17:35:59.221225+0000","last_unstale":"2024-05-04T17:39:59.459303+0000","last_undegraded":"2024-05-04T17:39:59.459303+0000","last_fullsized":"2024-05-04T17:39:59.459303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:39:59.459219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:39:59.459219+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.459219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:19:46.872310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'6197","reported_seq":6229,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:56.915187+0000","last_change":"2024-05-04T17:39:56.915259+0000","last_active":"2024-05-04T17:39:56.915187+0000","last_peered":"2024-05-04T17:39:56.915187+0000","last_clean":"2024-05-04T17:39:56.915187+0000","last_became_active":"2024-05-04T17:36:05.247697+0000","last_became_peered":"2024-05-04T17:36:05.247697+0000","last_unstale":"2024-05-04T17:39:56.915187+0000","last_undegraded":"2024-05-04T17:39:56.915187+0000","last_fullsized":"2024-05-04T17:39:56.915187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6197","last_scrub_stamp":"2024-05-04T17:39:56.915112+0000","last_deep_scrub":"23'6197","last_deep_scrub_stamp":"2024-05-04T17:39:56.915112+0000","last_clean_scrub_stamp":"2024-05-04T17:39:56.915112+0000","objects_scrubbed":21,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:06:43.614325+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78672920,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":6162,"num_write_kb":192039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:58.389294+0000","last_change":"2024-05-04T17:39:58.389294+0000","last_active":"2024-05-04T17:39:58.389294+0000","last_peered":"2024-05-04T17:39:58.389294+0000","last_clean":"2024-05-04T17:39:58.389294+0000","last_became_active":"2024-05-04T17:35:59.221171+0000","last_became_peered":"2024-05-04T17:35:59.221171+0000","last_unstale":"2024-05-04T17:39:58.389294+0000","last_undegraded":"2024-05-04T17:39:58.389294+0000","last_fullsized":"2024-05-04T17:39:58.389294+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:39:58.389185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:39:58.389185+0000","last_clean_scrub_stamp":"2024-05-04T17:39:58.389185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:29:57.490857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'4012","reported_seq":4048,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:03.258904+0000","last_change":"2024-05-04T17:40:03.258946+0000","last_active":"2024-05-04T17:40:03.258904+0000","last_peered":"2024-05-04T17:40:03.258904+0000","last_clean":"2024-05-04T17:40:03.258904+0000","last_became_active":"2024-05-04T17:36:05.261828+0000","last_became_peered":"2024-05-04T17:36:05.261828+0000","last_unstale":"2024-05-04T17:40:03.258904+0000","last_undegraded":"2024-05-04T17:40:03.258904+0000","last_fullsized":"2024-05-04T17:40:03.258904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4012","last_scrub_stamp":"2024-05-04T17:40:03.258828+0000","last_deep_scrub":"23'4012","last_deep_scrub_stamp":"2024-05-04T17:40:03.258828+0000","last_clean_scrub_stamp":"2024-05-04T17:40:03.258828+0000","objects_scrubbed":26,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:17:25.068639+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":3961,"num_write_kb":185542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.839827+0000","last_change":"2024-05-04T17:40:00.839827+0000","last_active":"2024-05-04T17:40:00.839827+0000","last_peered":"2024-05-04T17:40:00.839827+0000","last_clean":"2024-05-04T17:40:00.839827+0000","last_became_active":"2024-05-04T17:35:59.221846+0000","last_became_peered":"2024-05-04T17:35:59.221846+0000","last_unstale":"2024-05-04T17:40:00.839827+0000","last_undegraded":"2024-05-04T17:40:00.839827+0000","last_fullsized":"2024-05-04T17:40:00.839827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:40:00.839713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:40:00.839713+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.839713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:59:22.218602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'5149","reported_seq":5178,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.447585+0000","last_change":"2024-05-04T17:40:00.447585+0000","last_active":"2024-05-04T17:40:00.447585+0000","last_peered":"2024-05-04T17:40:00.447585+0000","last_clean":"2024-05-04T17:40:00.447585+0000","last_became_active":"2024-05-04T17:36:05.395267+0000","last_became_peered":"2024-05-04T17:36:05.395267+0000","last_unstale":"2024-05-04T17:40:00.447585+0000","last_undegraded":"2024-05-04T17:40:00.447585+0000","last_fullsized":"2024-05-04T17:40:00.447585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5149","last_scrub_stamp":"2024-05-04T17:40:00.447440+0000","last_deep_scrub":"23'5149","last_deep_scrub_stamp":"2024-05-04T17:40:00.447440+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.447440+0000","objects_scrubbed":17,"log_size":5149,"log_dups_size":0,"ondisk_log_size":5149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:13:36.317050+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":103041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:57.345077+0000","last_change":"2024-05-04T17:39:57.345077+0000","last_active":"2024-05-04T17:39:57.345077+0000","last_peered":"2024-05-04T17:39:57.345077+0000","last_clean":"2024-05-04T17:39:57.345077+0000","last_became_active":"2024-05-04T17:35:59.218428+0000","last_became_peered":"2024-05-04T17:35:59.218428+0000","last_unstale":"2024-05-04T17:39:57.345077+0000","last_undegraded":"2024-05-04T17:39:57.345077+0000","last_fullsized":"2024-05-04T17:39:57.345077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:39:57.344988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:39:57.344988+0000","last_clean_scrub_stamp":"2024-05-04T17:39:57.344988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:04:02.687545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'6501","reported_seq":6561,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.557514+0000","last_change":"2024-05-04T17:40:01.557553+0000","last_active":"2024-05-04T17:40:01.557514+0000","last_peered":"2024-05-04T17:40:01.557514+0000","last_clean":"2024-05-04T17:40:01.557514+0000","last_became_active":"2024-05-04T17:36:05.254049+0000","last_became_peered":"2024-05-04T17:36:05.254049+0000","last_unstale":"2024-05-04T17:40:01.557514+0000","last_undegraded":"2024-05-04T17:40:01.557514+0000","last_fullsized":"2024-05-04T17:40:01.557514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6501","last_scrub_stamp":"2024-05-04T17:40:01.557444+0000","last_deep_scrub":"23'6501","last_deep_scrub_stamp":"2024-05-04T17:40:01.557444+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.557444+0000","objects_scrubbed":31,"log_size":6501,"log_dups_size":0,"ondisk_log_size":6501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:40:13.678708+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123508560,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":117,"num_read_kb":157,"num_write":6516,"num_write_kb":216324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'4412","reported_seq":4447,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.939867+0000","last_change":"2024-05-04T17:39:59.939867+0000","last_active":"2024-05-04T17:39:59.939867+0000","last_peered":"2024-05-04T17:39:59.939867+0000","last_clean":"2024-05-04T17:39:59.939867+0000","last_became_active":"2024-05-04T17:36:05.259684+0000","last_became_peered":"2024-05-04T17:36:05.259684+0000","last_unstale":"2024-05-04T17:39:59.939867+0000","last_undegraded":"2024-05-04T17:39:59.939867+0000","last_fullsized":"2024-05-04T17:39:59.939867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4412","last_scrub_stamp":"2024-05-04T17:39:59.939767+0000","last_deep_scrub":"23'4412","last_deep_scrub_stamp":"2024-05-04T17:39:59.939767+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.939767+0000","objects_scrubbed":32,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:15:33.234309+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829632,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":4394,"num_write_kb":197895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.273820+0000","last_change":"2024-05-04T17:40:00.273820+0000","last_active":"2024-05-04T17:40:00.273820+0000","last_peered":"2024-05-04T17:40:00.273820+0000","last_clean":"2024-05-04T17:40:00.273820+0000","last_became_active":"2024-05-04T17:35:59.222812+0000","last_became_peered":"2024-05-04T17:35:59.222812+0000","last_unstale":"2024-05-04T17:40:00.273820+0000","last_undegraded":"2024-05-04T17:40:00.273820+0000","last_fullsized":"2024-05-04T17:40:00.273820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:40:00.273729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:40:00.273729+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.273729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:34:06.051884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"23'3993","reported_seq":4023,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.480639+0000","last_change":"2024-05-04T17:40:01.480639+0000","last_active":"2024-05-04T17:40:01.480639+0000","last_peered":"2024-05-04T17:40:01.480639+0000","last_clean":"2024-05-04T17:40:01.480639+0000","last_became_active":"2024-05-04T17:36:05.395272+0000","last_became_peered":"2024-05-04T17:36:05.395272+0000","last_unstale":"2024-05-04T17:40:01.480639+0000","last_undegraded":"2024-05-04T17:40:01.480639+0000","last_fullsized":"2024-05-04T17:40:01.480639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3993","last_scrub_stamp":"2024-05-04T17:40:01.480509+0000","last_deep_scrub":"23'3993","last_deep_scrub_stamp":"2024-05-04T17:40:01.480509+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.480509+0000","objects_scrubbed":27,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:46:45.708006+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110366368,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":158385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:57.341555+0000","last_change":"2024-05-04T17:39:57.341555+0000","last_active":"2024-05-04T17:39:57.341555+0000","last_peered":"2024-05-04T17:39:57.341555+0000","last_clean":"2024-05-04T17:39:57.341555+0000","last_became_active":"2024-05-04T17:35:59.331054+0000","last_became_peered":"2024-05-04T17:35:59.331054+0000","last_unstale":"2024-05-04T17:39:57.341555+0000","last_undegraded":"2024-05-04T17:39:57.341555+0000","last_fullsized":"2024-05-04T17:39:57.341555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:39:57.341452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:39:57.341452+0000","last_clean_scrub_stamp":"2024-05-04T17:39:57.341452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:07:27.943781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"23'4127","reported_seq":4157,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.973175+0000","last_change":"2024-05-04T17:40:01.973175+0000","last_active":"2024-05-04T17:40:01.973175+0000","last_peered":"2024-05-04T17:40:01.973175+0000","last_clean":"2024-05-04T17:40:01.973175+0000","last_became_active":"2024-05-04T17:36:05.261395+0000","last_became_peered":"2024-05-04T17:36:05.261395+0000","last_unstale":"2024-05-04T17:40:01.973175+0000","last_undegraded":"2024-05-04T17:40:01.973175+0000","last_fullsized":"2024-05-04T17:40:01.973175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4127","last_scrub_stamp":"2024-05-04T17:40:01.973082+0000","last_deep_scrub":"23'4127","last_deep_scrub_stamp":"2024-05-04T17:40:01.973082+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.973082+0000","objects_scrubbed":26,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:41:02.913994+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106299392,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":186884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.832722+0000","last_change":"2024-05-04T17:40:01.832722+0000","last_active":"2024-05-04T17:40:01.832722+0000","last_peered":"2024-05-04T17:40:01.832722+0000","last_clean":"2024-05-04T17:40:01.832722+0000","last_became_active":"2024-05-04T17:35:59.223087+0000","last_became_peered":"2024-05-04T17:35:59.223087+0000","last_unstale":"2024-05-04T17:40:01.832722+0000","last_undegraded":"2024-05-04T17:40:01.832722+0000","last_fullsized":"2024-05-04T17:40:01.832722+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:40:01.832591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:40:01.832591+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.832591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:35:45.717414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"23'5791","reported_seq":5821,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:02.301297+0000","last_change":"2024-05-04T17:40:02.301297+0000","last_active":"2024-05-04T17:40:02.301297+0000","last_peered":"2024-05-04T17:40:02.301297+0000","last_clean":"2024-05-04T17:40:02.301297+0000","last_became_active":"2024-05-04T17:36:05.257750+0000","last_became_peered":"2024-05-04T17:36:05.257750+0000","last_unstale":"2024-05-04T17:40:02.301297+0000","last_undegraded":"2024-05-04T17:40:02.301297+0000","last_fullsized":"2024-05-04T17:40:02.301297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5791","last_scrub_stamp":"2024-05-04T17:40:02.301162+0000","last_deep_scrub":"23'5791","last_deep_scrub_stamp":"2024-05-04T17:40:02.301162+0000","last_clean_scrub_stamp":"2024-05-04T17:40:02.301162+0000","objects_scrubbed":29,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:17:46.511266+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":181199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.295597+0000","last_change":"2024-05-04T17:39:59.295597+0000","last_active":"2024-05-04T17:39:59.295597+0000","last_peered":"2024-05-04T17:39:59.295597+0000","last_clean":"2024-05-04T17:39:59.295597+0000","last_became_active":"2024-05-04T17:35:59.221645+0000","last_became_peered":"2024-05-04T17:35:59.221645+0000","last_unstale":"2024-05-04T17:39:59.295597+0000","last_undegraded":"2024-05-04T17:39:59.295597+0000","last_fullsized":"2024-05-04T17:39:59.295597+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:39:59.295502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:39:59.295502+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.295502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:01:18.862015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'119","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.825670+0000","last_change":"2024-05-04T17:39:59.825701+0000","last_active":"2024-05-04T17:39:59.825670+0000","last_peered":"2024-05-04T17:39:59.825670+0000","last_clean":"2024-05-04T17:39:59.825670+0000","last_became_active":"2024-05-04T17:35:58.172416+0000","last_became_peered":"2024-05-04T17:35:58.172416+0000","last_unstale":"2024-05-04T17:39:59.825670+0000","last_undegraded":"2024-05-04T17:39:59.825670+0000","last_fullsized":"2024-05-04T17:39:59.825670+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'119","last_scrub_stamp":"2024-05-04T17:39:59.825609+0000","last_deep_scrub":"23'119","last_deep_scrub_stamp":"2024-05-04T17:39:59.825609+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.825609+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:20:57.546647+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"23'2756","reported_seq":2795,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:59.163996+0000","last_change":"2024-05-04T17:39:59.164062+0000","last_active":"2024-05-04T17:39:59.163996+0000","last_peered":"2024-05-04T17:39:59.163996+0000","last_clean":"2024-05-04T17:39:59.163996+0000","last_became_active":"2024-05-04T17:36:05.249926+0000","last_became_peered":"2024-05-04T17:36:05.249926+0000","last_unstale":"2024-05-04T17:39:59.163996+0000","last_undegraded":"2024-05-04T17:39:59.163996+0000","last_fullsized":"2024-05-04T17:39:59.163996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2756","last_scrub_stamp":"2024-05-04T17:39:59.163925+0000","last_deep_scrub":"23'2756","last_deep_scrub_stamp":"2024-05-04T17:39:59.163925+0000","last_clean_scrub_stamp":"2024-05-04T17:39:59.163925+0000","objects_scrubbed":23,"log_size":2756,"log_dups_size":0,"ondisk_log_size":2756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:55:20.404932+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":2682,"num_write_kb":127745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"23'9973","reported_seq":10125,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.968370+0000","last_change":"2024-05-04T17:40:01.968400+0000","last_active":"2024-05-04T17:40:01.968370+0000","last_peered":"2024-05-04T17:40:01.968370+0000","last_clean":"2024-05-04T17:40:01.968370+0000","last_became_active":"2024-05-04T17:36:05.395416+0000","last_became_peered":"2024-05-04T17:36:05.395416+0000","last_unstale":"2024-05-04T17:40:01.968370+0000","last_undegraded":"2024-05-04T17:40:01.968370+0000","last_fullsized":"2024-05-04T17:40:01.968370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'9973","last_scrub_stamp":"2024-05-04T17:40:01.968295+0000","last_deep_scrub":"23'9973","last_deep_scrub_stamp":"2024-05-04T17:40:01.968295+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.968295+0000","objects_scrubbed":28,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:50:44.953880+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":386,"num_read_kb":352,"num_write":6860,"num_write_kb":187028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"23'4878","reported_seq":4910,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:01.786482+0000","last_change":"2024-05-04T17:40:01.786538+0000","last_active":"2024-05-04T17:40:01.786482+0000","last_peered":"2024-05-04T17:40:01.786482+0000","last_clean":"2024-05-04T17:40:01.786482+0000","last_became_active":"2024-05-04T17:36:05.257078+0000","last_became_peered":"2024-05-04T17:36:05.257078+0000","last_unstale":"2024-05-04T17:40:01.786482+0000","last_undegraded":"2024-05-04T17:40:01.786482+0000","last_fullsized":"2024-05-04T17:40:01.786482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4878","last_scrub_stamp":"2024-05-04T17:40:01.786384+0000","last_deep_scrub":"23'4878","last_deep_scrub_stamp":"2024-05-04T17:40:01.786384+0000","last_clean_scrub_stamp":"2024-05-04T17:40:01.786384+0000","objects_scrubbed":31,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:46:47.499994+0000","scrub_duration":339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4826,"num_write_kb":165126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"23'4372","reported_seq":4404,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:39:58.238817+0000","last_change":"2024-05-04T17:39:58.238861+0000","last_active":"2024-05-04T17:39:58.238817+0000","last_peered":"2024-05-04T17:39:58.238817+0000","last_clean":"2024-05-04T17:39:58.238817+0000","last_became_active":"2024-05-04T17:36:05.249845+0000","last_became_peered":"2024-05-04T17:36:05.249845+0000","last_unstale":"2024-05-04T17:39:58.238817+0000","last_undegraded":"2024-05-04T17:39:58.238817+0000","last_fullsized":"2024-05-04T17:39:58.238817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4372","last_scrub_stamp":"2024-05-04T17:39:58.238753+0000","last_deep_scrub":"23'4372","last_deep_scrub_stamp":"2024-05-04T17:39:58.238753+0000","last_clean_scrub_stamp":"2024-05-04T17:39:58.238753+0000","objects_scrubbed":32,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:46.756248+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":4449,"num_write_kb":193090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"23'5698","reported_seq":5852,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:02.582760+0000","last_change":"2024-05-04T17:40:02.582798+0000","last_active":"2024-05-04T17:40:02.582760+0000","last_peered":"2024-05-04T17:40:02.582760+0000","last_clean":"2024-05-04T17:40:02.582760+0000","last_became_active":"2024-05-04T17:36:05.257760+0000","last_became_peered":"2024-05-04T17:36:05.257760+0000","last_unstale":"2024-05-04T17:40:02.582760+0000","last_undegraded":"2024-05-04T17:40:02.582760+0000","last_fullsized":"2024-05-04T17:40:02.582760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5698","last_scrub_stamp":"2024-05-04T17:40:02.582703+0000","last_deep_scrub":"23'5698","last_deep_scrub_stamp":"2024-05-04T17:40:02.582703+0000","last_clean_scrub_stamp":"2024-05-04T17:40:02.582703+0000","objects_scrubbed":36,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:49:03.892886+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140053176,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":196,"num_read_kb":182,"num_write":5664,"num_write_kb":217362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"23'3560","reported_seq":3606,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:04.009643+0000","last_change":"2024-05-04T17:40:04.009687+0000","last_active":"2024-05-04T17:40:04.009643+0000","last_peered":"2024-05-04T17:40:04.009643+0000","last_clean":"2024-05-04T17:40:04.009643+0000","last_became_active":"2024-05-04T17:36:05.261259+0000","last_became_peered":"2024-05-04T17:36:05.261259+0000","last_unstale":"2024-05-04T17:40:04.009643+0000","last_undegraded":"2024-05-04T17:40:04.009643+0000","last_fullsized":"2024-05-04T17:40:04.009643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3560","last_scrub_stamp":"2024-05-04T17:40:04.009549+0000","last_deep_scrub":"23'3560","last_deep_scrub_stamp":"2024-05-04T17:40:04.009549+0000","last_clean_scrub_stamp":"2024-05-04T17:40:04.009549+0000","objects_scrubbed":28,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:53:32.561012+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":66,"num_read_kb":115,"num_write":3541,"num_write_kb":181187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"23'4882","reported_seq":5003,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:08.104827+0000","last_change":"2024-05-04T17:40:08.104859+0000","last_active":"2024-05-04T17:40:08.104827+0000","last_peered":"2024-05-04T17:40:08.104827+0000","last_clean":"2024-05-04T17:40:08.104827+0000","last_became_active":"2024-05-04T17:36:05.263214+0000","last_became_peered":"2024-05-04T17:36:05.263214+0000","last_unstale":"2024-05-04T17:40:08.104827+0000","last_undegraded":"2024-05-04T17:40:08.104827+0000","last_fullsized":"2024-05-04T17:40:08.104827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4882","last_scrub_stamp":"2024-05-04T17:40:08.104758+0000","last_deep_scrub":"23'4882","last_deep_scrub_stamp":"2024-05-04T17:40:08.104758+0000","last_clean_scrub_stamp":"2024-05-04T17:40:08.104758+0000","objects_scrubbed":29,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:45:01.329264+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":300,"num_read_kb":235,"num_write":5104,"num_write_kb":162415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"23'3538","reported_seq":3566,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:06.000688+0000","last_change":"2024-05-04T17:40:06.000741+0000","last_active":"2024-05-04T17:40:06.000688+0000","last_peered":"2024-05-04T17:40:06.000688+0000","last_clean":"2024-05-04T17:40:06.000688+0000","last_became_active":"2024-05-04T17:36:05.262817+0000","last_became_peered":"2024-05-04T17:36:05.262817+0000","last_unstale":"2024-05-04T17:40:06.000688+0000","last_undegraded":"2024-05-04T17:40:06.000688+0000","last_fullsized":"2024-05-04T17:40:06.000688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3538","last_scrub_stamp":"2024-05-04T17:40:06.000622+0000","last_deep_scrub":"23'3538","last_deep_scrub_stamp":"2024-05-04T17:40:06.000622+0000","last_clean_scrub_stamp":"2024-05-04T17:40:06.000622+0000","objects_scrubbed":20,"log_size":3538,"log_dups_size":0,"ondisk_log_size":3538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:55:07.179182+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80883712,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":134410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"23'3839","reported_seq":3870,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.665763+0000","last_change":"2024-05-04T17:40:00.665817+0000","last_active":"2024-05-04T17:40:00.665763+0000","last_peered":"2024-05-04T17:40:00.665763+0000","last_clean":"2024-05-04T17:40:00.665763+0000","last_became_active":"2024-05-04T17:36:05.249932+0000","last_became_peered":"2024-05-04T17:36:05.249932+0000","last_unstale":"2024-05-04T17:40:00.665763+0000","last_undegraded":"2024-05-04T17:40:00.665763+0000","last_fullsized":"2024-05-04T17:40:00.665763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3839","last_scrub_stamp":"2024-05-04T17:40:00.665652+0000","last_deep_scrub":"23'3839","last_deep_scrub_stamp":"2024-05-04T17:40:00.665652+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.665652+0000","objects_scrubbed":28,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:58:19.085568+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112574992,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3768,"num_write_kb":163450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"23'3935","reported_seq":3971,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:07.022720+0000","last_change":"2024-05-04T17:40:07.022746+0000","last_active":"2024-05-04T17:40:07.022720+0000","last_peered":"2024-05-04T17:40:07.022720+0000","last_clean":"2024-05-04T17:40:07.022720+0000","last_became_active":"2024-05-04T17:36:05.263000+0000","last_became_peered":"2024-05-04T17:36:05.263000+0000","last_unstale":"2024-05-04T17:40:07.022720+0000","last_undegraded":"2024-05-04T17:40:07.022720+0000","last_fullsized":"2024-05-04T17:40:07.022720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3935","last_scrub_stamp":"2024-05-04T17:40:07.022653+0000","last_deep_scrub":"23'3935","last_deep_scrub_stamp":"2024-05-04T17:40:07.022653+0000","last_clean_scrub_stamp":"2024-05-04T17:40:07.022653+0000","objects_scrubbed":26,"log_size":3935,"log_dups_size":0,"ondisk_log_size":3935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:50.512705+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":3923,"num_write_kb":150485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"23'3460","reported_seq":3502,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:00.960317+0000","last_change":"2024-05-04T17:40:00.960360+0000","last_active":"2024-05-04T17:40:00.960317+0000","last_peered":"2024-05-04T17:40:00.960317+0000","last_clean":"2024-05-04T17:40:00.960317+0000","last_became_active":"2024-05-04T17:36:05.260881+0000","last_became_peered":"2024-05-04T17:36:05.260881+0000","last_unstale":"2024-05-04T17:40:00.960317+0000","last_undegraded":"2024-05-04T17:40:00.960317+0000","last_fullsized":"2024-05-04T17:40:00.960317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3460","last_scrub_stamp":"2024-05-04T17:40:00.960254+0000","last_deep_scrub":"23'3460","last_deep_scrub_stamp":"2024-05-04T17:40:00.960254+0000","last_clean_scrub_stamp":"2024-05-04T17:40:00.960254+0000","objects_scrubbed":31,"log_size":3460,"log_dups_size":0,"ondisk_log_size":3460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:09:16.450880+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":54,"num_read_kb":41,"num_write":3408,"num_write_kb":117896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"23'8275","reported_seq":8307,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-04T17:40:05.128674+0000","last_change":"2024-05-04T17:40:05.128736+0000","last_active":"2024-05-04T17:40:05.128674+0000","last_peered":"2024-05-04T17:40:05.128674+0000","last_clean":"2024-05-04T17:40:05.128674+0000","last_became_active":"2024-05-04T17:36:05.262502+0000","last_became_peered":"2024-05-04T17:36:05.262502+0000","last_unstale":"2024-05-04T17:40:05.128674+0000","last_undegraded":"2024-05-04T17:40:05.128674+0000","last_fullsized":"2024-05-04T17:40:05.128674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8275","last_scrub_stamp":"2024-05-04T17:40:05.128581+0000","last_deep_scrub":"23'8275","last_deep_scrub_stamp":"2024-05-04T17:40:05.128581+0000","last_clean_scrub_stamp":"2024-05-04T17:40:05.128581+0000","objects_scrubbed":33,"log_size":8275,"log_dups_size":0,"ondisk_log_size":8275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:49:54.753974+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8220,"num_write_kb":197892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3578536184,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":2570,"num_read_kb":2586,"num_write":163631,"num_write_kb":5652201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6859538432,"data_stored":6855271544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167384,"ondisk_log_size":167384,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542198,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343508,"kb_used_data":1055564,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93028332,"statfs":{"total":96636764160,"available":95261011968,"internally_reserved":0,"allocated":1080897536,"data_stored":1080007567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":294844874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542199,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1270788,"kb_used_data":1005052,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":93101052,"statfs":{"total":96636764160,"available":95335477248,"internally_reserved":0,"allocated":1029173248,"data_stored":1028426019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272103877},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542200,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1728892,"kb_used_data":1350772,"kb_used_omap":1,"kb_used_meta":378110,"kb_avail":92642948,"statfs":{"total":96636764160,"available":94866378752,"internally_reserved":0,"allocated":1383190528,"data_stored":1382086063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387185093},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401624,"kb_used_data":330256,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93970216,"statfs":{"total":96636764160,"available":96225501184,"internally_reserved":0,"allocated":338182144,"data_stored":337870963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362148,"kb_used_data":1074204,"kb_used_omap":1,"kb_used_meta":287934,"kb_avail":93009692,"statfs":{"total":96636764160,"available":95241924608,"internally_reserved":0,"allocated":1099984896,"data_stored":1099156671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294844871},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377680,"kb_used_data":288200,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93994160,"statfs":{"total":96636764160,"available":96250019840,"internally_reserved":0,"allocated":295116800,"data_stored":294765651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649120,"kb_used_data":545816,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93722720,"statfs":{"total":96636764160,"available":95972065280,"internally_reserved":0,"allocated":558915584,"data_stored":558501959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1317408,"kb_used_data":1051672,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":93054432,"statfs":{"total":96636764160,"available":95287738368,"internally_reserved":0,"allocated":1076912128,"data_stored":1076111395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272103877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1076158464,"data_stored":1075507856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":558694400,"data_stored":558427348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294895616,"data_stored":294691040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1099763712,"data_stored":1099082060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1382969344,"data_stored":1382011452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1028952064,"data_stored":1028351408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":337960960,"data_stored":337796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080143872,"data_stored":1079404028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:40:20.024 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-04T17:40:20.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.120+0000 7f4dff1db640 1 Processor -- start 2024-05-04T17:40:20.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.120+0000 7f4dff1db640 1 -- start start 2024-05-04T17:40:20.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4dff1db640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df80a9b30 con 0x7f4df80a9750 2024-05-04T17:40:20.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4dff1db640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df80ab460 con 0x7f4df813d780 2024-05-04T17:40:20.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4dff1db640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df80b1e20 con 0x7f4df80aa1e0 2024-05-04T17:40:20.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4dfcf77640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4df80a9750 0x7f4df806f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53380/0 (socket says 172.21.15.110:53380) 2024-05-04T17:40:20.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4dfcf77640 1 -- 172.21.15.110:0/295914245 learned_addr learned my addr 172.21.15.110:0/295914245 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:40:20.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df7fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f4df80aa1e0 0x7f4df8140f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33858/0 (socket says 172.21.15.110:33858) 2024-05-04T17:40:20.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 982135353 0 0) 0x7f4df80a9b30 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4de0003520 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321144820 0 0) 0x7f4df80b1e20 con 0x7f4df80aa1e0 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df80a9b30 con 0x7f4df80aa1e0 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2658308402 0 0) 0x7f4de0003520 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 >> v1:172.21.15.110:6790/0 conn(0x7f4df80aa1e0 legacy=0x7f4df8140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 >> v1:172.21.15.136:6789/0 conn(0x7f4df813d780 legacy=0x7f4df814bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df8140630 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4de8001610 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4de8001870 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.121+0000 7f4df77fe640 1 -- 172.21.15.110:0/295914245 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4de8001c80 con 0x7f4df80a9750 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.122+0000 7f4dff1db640 1 -- 172.21.15.110:0/295914245 >> v1:172.21.15.110:6789/0 conn(0x7f4df80a9750 legacy=0x7f4df806f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.122+0000 7f4dff1db640 1 -- 172.21.15.110:0/295914245 shutdown_connections 2024-05-04T17:40:20.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.122+0000 7f4dff1db640 1 -- 172.21.15.110:0/295914245 >> 172.21.15.110:0/295914245 conn(0x7f4df80b0e30 msgr2=0x7f4df80b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:40:20.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.122+0000 7f4dff1db640 1 -- 172.21.15.110:0/295914245 shutdown_connections 2024-05-04T17:40:20.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.122+0000 7f4dff1db640 1 -- 172.21.15.110:0/295914245 wait complete. 2024-05-04T17:40:20.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.123+0000 7f4dff1db640 1 Processor -- start 2024-05-04T17:40:20.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.123+0000 7f4dff1db640 1 -- start start 2024-05-04T17:40:20.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4dff1db640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df8142390 con 0x7f4df80aa1e0 2024-05-04T17:40:20.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4dff1db640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df81449c0 con 0x7f4df813d780 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4dff1db640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df8142e90 con 0x7f4df80a9750 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df7fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4df80aa1e0 0x7f4df8136490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53388/0 (socket says 172.21.15.110:53388) 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df7fff640 1 -- 172.21.15.110:0/1888221776 learned_addr learned my addr 172.21.15.110:0/1888221776 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4dfcf77640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f4df80a9750 0x7f4df81dac00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33864/0 (socket says 172.21.15.110:33864) 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475262275 0 0) 0x7f4df8142e90 con 0x7f4df80a9750 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4de4003520 con 0x7f4df80a9750 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 42444183 0 0) 0x7f4df8142390 con 0x7f4df80aa1e0 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df8142e90 con 0x7f4df80aa1e0 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 492900756 0 0) 0x7f4df81449c0 con 0x7f4df813d780 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df8142390 con 0x7f4df813d780 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2957922898 0 0) 0x7f4df8142e90 con 0x7f4df80aa1e0 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4df81449c0 con 0x7f4df80aa1e0 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1556174563 0 0) 0x7f4de4003520 con 0x7f4df80a9750 2024-05-04T17:40:20.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4df8142e90 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4dec001360 con 0x7f4df80aa1e0 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4de8001bc0 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 538947626 0 0) 0x7f4df8142390 con 0x7f4df813d780 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4de4003520 con 0x7f4df813d780 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4df0001a50 con 0x7f4df813d780 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2603587399 0 0) 0x7f4df8142e90 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 >> v1:172.21.15.136:6789/0 conn(0x7f4df813d780 legacy=0x7f4df81e2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 >> v1:172.21.15.110:6789/0 conn(0x7f4df80aa1e0 legacy=0x7f4df8136490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df81445e0 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.124+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4df8072d90 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.125+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4df81db4f0 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.125+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4de8004c60 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.125+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3440346412 0 0) 0x7f4de8004e10 con 0x7f4df80a9750 2024-05-04T17:40:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.126+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd8005080 con 0x7f4df80a9750 2024-05-04T17:40:20.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.126+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 5) ==== 90463+0+0 (unknown 1877082340 0 0) 0x7f4de8004fb0 con 0x7f4df80a9750 2024-05-04T17:40:20.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.126+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 4525+0+0 (unknown 3699820440 0 0) 0x7f4de8086310 con 0x7f4df80a9750 2024-05-04T17:40:20.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.131+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (unknown 1092875540 0 2154935188) 0x7f4de8055030 con 0x7f4df80a9750 2024-05-04T17:40:20.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.303+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 --> v1:172.21.15.110:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4dd8005370 con 0x7f4df80a9750 2024-05-04T17:40:20.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.307+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f4de8001200 con 0x7f4df80a9750 2024-05-04T17:40:20.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.307+0000 7f4df57fa640 1 -- 172.21.15.110:0/1888221776 <== mon.2 v1:172.21.15.110:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (unknown 1453024670 0 0) 0x7f4de8054930 con 0x7f4df80a9750 2024-05-04T17:40:20.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.309+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 >> v1:172.21.15.110:6816/783516658 conn(0x7f4de406ca70 legacy=0x7f4de406ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.309+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 >> v1:172.21.15.110:6790/0 conn(0x7f4df80a9750 legacy=0x7f4df81dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:40:20.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.310+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 shutdown_connections 2024-05-04T17:40:20.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.310+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 >> 172.21.15.110:0/1888221776 conn(0x7f4df80b0e30 msgr2=0x7f4df80b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:40:20.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.310+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 shutdown_connections 2024-05-04T17:40:20.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-04T17:40:20.310+0000 7f4dff1db640 1 -- 172.21.15.110:0/1888221776 wait complete. 2024-05-04T17:40:20.320 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-04T17:40:20.321 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-04T17:40:20.321 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-04T17:40:20.321 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.373 INFO:tasks.ceph.osd.0:Stopped 2024-05-04T17:40:20.373 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-04T17:40:20.373 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.424 INFO:tasks.ceph.osd.1:Stopped 2024-05-04T17:40:20.425 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-04T17:40:20.425 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.465 INFO:tasks.ceph.osd.2:Stopped 2024-05-04T17:40:20.466 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-04T17:40:20.466 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.509 INFO:tasks.ceph.osd.3:Stopped 2024-05-04T17:40:20.510 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-04T17:40:20.510 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.562 INFO:tasks.ceph.osd.4:Stopped 2024-05-04T17:40:20.563 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-04T17:40:20.563 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.613 INFO:tasks.ceph.osd.5:Stopped 2024-05-04T17:40:20.614 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-04T17:40:20.614 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.665 INFO:tasks.ceph.osd.6:Stopped 2024-05-04T17:40:20.665 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-04T17:40:20.665 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.720 INFO:tasks.ceph.osd.7:Stopped 2024-05-04T17:40:20.720 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-04T17:40:20.720 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-04T17:40:20.720 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.744 INFO:tasks.ceph.mgr.y:Stopped 2024-05-04T17:40:20.745 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-04T17:40:20.745 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.771 INFO:tasks.ceph.mgr.x:Stopped 2024-05-04T17:40:20.771 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-04T17:40:20.771 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-04T17:40:20.771 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.796 INFO:tasks.ceph.mon.a:Stopped 2024-05-04T17:40:20.796 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-04T17:40:20.796 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.846 INFO:tasks.ceph.mon.c:Stopped 2024-05-04T17:40:20.847 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-04T17:40:20.847 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T17:40:20.858 INFO:tasks.ceph.mon.b:Stopped 2024-05-04T17:40:20.859 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-04T17:40:20.859 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | head -n 1 2024-05-04T17:40:20.912 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:40:20.912 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-04T17:40:21.696 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:40:21.696 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-04T17:40:21.784 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:40:21.784 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-04T17:40:21.845 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:40:21.845 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-04T17:40:21.946 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:40:21.947 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-04T17:40:22.300 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:40:22.301 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-04T17:40:22.383 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:40:22.383 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-04T17:40:22.438 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:40:22.438 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-04T17:40:22.498 INFO:tasks.ceph:Archiving mon data... 2024-05-04T17:40:22.499 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/data/mon.a.tgz 2024-05-04T17:40:22.499 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-05-04T17:40:22.515 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.ibmtTvm70q 2024-05-04T17:40:22.515 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ibmtTvm70q 2024-05-04T17:40:22.635 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.ibmtTvm70q 2024-05-04T17:40:22.719 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.ibmtTvm70q is 277KB 2024-05-04T17:40:22.815 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.ibmtTvm70q 2024-05-04T17:40:22.829 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/data/mon.c.tgz 2024-05-04T17:40:22.829 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-05-04T17:40:22.884 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.NBn8JDSa9E 2024-05-04T17:40:22.884 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.NBn8JDSa9E 2024-05-04T17:40:22.992 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.NBn8JDSa9E 2024-05-04T17:40:23.074 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.NBn8JDSa9E is 278KB 2024-05-04T17:40:23.169 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.NBn8JDSa9E 2024-05-04T17:40:23.184 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/data/mon.b.tgz 2024-05-04T17:40:23.184 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-05-04T17:40:23.201 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.GiOqfZQ7MU 2024-05-04T17:40:23.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.GiOqfZQ7MU 2024-05-04T17:40:23.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.GiOqfZQ7MU 2024-05-04T17:40:23.419 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.GiOqfZQ7MU is 278KB 2024-05-04T17:40:23.514 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.GiOqfZQ7MU 2024-05-04T17:40:23.528 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-04T17:40:23.528 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-04T17:40:23.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-04T17:40:23.610 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-04T17:40:23.649 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-04T17:40:23.649 INFO:tasks.ceph:Archiving crash dumps... 2024-05-04T17:40:23.651 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/remote/smithi110/crash 2024-05-04T17:40:23.651 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-04T17:40:23.675 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/remote/smithi136/crash 2024-05-04T17:40:23.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-04T17:40:23.711 INFO:tasks.ceph:Compressing logs... 2024-05-04T17:40:23.711 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-04T17:40:23.720 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-04T17:40:23.758 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36430.log 2024-05-04T17:40:23.758 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35771.log 2024-05-04T17:40:23.758 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35370.log 2024-05-04T17:40:23.758 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36763.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35370.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35771.log.gz 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30290.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.35370.log.gz 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.35894.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36763.log: /var/log/ceph/ceph-client.admin.30290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34595.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.36763.log.gz 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.30290.log.gz 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37057.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.35894.log: /var/log/ceph/ceph-client.admin.34595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35485.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37057.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.34595.log.gz 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.37057.log.gz 2024-05-04T17:40:23.759 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37435.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35485.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37519.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.35485.log.gz 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36889.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37435.log.gz 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37519.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36889.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37519.log.gz 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36573.log 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36889.log.gz 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35371.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35155.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-05-04T17:40:23.760 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5%/var/log/ceph/ceph-client.admin.35155.log: -- replaced with /var/log/ceph/ceph-client.admin.35371.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36721.log 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35155.log.gz 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37099.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.24124.log 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35489.log 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36721.log: /var/log/ceph/ceph.tmp-client.admin.24124.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.24124.log.gz 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35489.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.36721.log.gz 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36482.log 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35489.log.gz 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2024-05-04T17:40:23.761 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34815.log 2024-05-04T17:40:23.762 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35148.log: 88.1% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36482.log.gz 2024-05-04T17:40:23.762 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34814.log 2024-05-04T17:40:23.762 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-04T17:40:23.762 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz 2024-05-04T17:40:23.762 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34815.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.34815.log.gz 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34816.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.34814.log.gz 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37015.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36430.log.gz 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.34816.log.gz 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34453.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37015.log.gz 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34819.log: gzip -5 86.0% -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35153.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34453.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-04T17:40:23.763 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34453.log.gz 2024-05-04T17:40:23.764 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34817.log 2024-05-04T17:40:23.764 INFO:teuthology.orchestra.run.smithi110.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.35153.log.gz 2024-05-04T17:40:23.764 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-04T17:40:23.764 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34817.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-04T17:40:23.764 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35145.log 2024-05-04T17:40:23.764 INFO:teuthology.orchestra.run.smithi110.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.34817.log.gz 2024-05-04T17:40:23.768 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37141.log 2024-05-04T17:40:23.769 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35145.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.35145.log.gz 2024-05-04T17:40:23.769 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37267.log 2024-05-04T17:40:23.769 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37141.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37141.log.gz 2024-05-04T17:40:23.769 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36531.log 2024-05-04T17:40:23.769 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37267.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37267.log.gz 2024-05-04T17:40:23.769 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34818.log 2024-05-04T17:40:23.775 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37644.log 2024-05-04T17:40:23.776 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34818.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.34818.log.gz 2024-05-04T17:40:23.795 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30103.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29811.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30103.log.gz 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29852.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.29811.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29811.log.gz 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.29852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29983.log 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-04T17:40:23.796 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29852.log.gz 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.29983.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.29983.log.gz 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30063.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.30063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29936.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30063.log.gz 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5/var/log/ceph/ceph-client.admin.29936.log: --verbose -- /var/log/ceph/ceph-client.admin.29892.log 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29936.log.gz 2024-05-04T17:40:23.797 INFO:teuthology.orchestra.run.smithi136.stderr: -- /var/log/ceph/ceph-client.admin.30023.log 2024-05-04T17:40:23.798 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-04T17:40:23.798 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30023.log: /var/log/ceph/ceph-client.admin.29892.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29892.log.gz 2024-05-04T17:40:23.798 INFO:teuthology.orchestra.run.smithi136.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30023.log.gz 2024-05-04T17:40:23.798 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.b.log: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-04T17:40:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35152.log 2024-05-04T17:40:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37644.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37644.log.gz 2024-05-04T17:40:23.801 INFO:teuthology.orchestra.run.smithi110.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36531.log.gz 2024-05-04T17:40:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37604.log 2024-05-04T17:40:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35152.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35152.log.gz 2024-05-04T17:40:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-04T17:40:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37604.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37604.log.gz 2024-05-04T17:40:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36847.log 2024-05-04T17:40:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35490.log 2024-05-04T17:40:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36847.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36847.log.gz 2024-05-04T17:40:23.820 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35729.log 2024-05-04T17:40:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35490.log: 90.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-04T17:40:23.820 INFO:teuthology.orchestra.run.smithi110.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.35490.log.gz 2024-05-04T17:40:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-05-04T17:40:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35729.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35729.log.gz 2024-05-04T17:40:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-04T17:40:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35488.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-05-04T17:40:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36805.log 2024-05-04T17:40:23.828 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37351.log 2024-05-04T17:40:23.829 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36805.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.36805.log.gz 2024-05-04T17:40:23.843 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37225.log 2024-05-04T17:40:23.843 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37351.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37351.log.gz 2024-05-04T17:40:23.852 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-05-04T17:40:23.865 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37225.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37225.log.gz 2024-05-04T17:40:23.868 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35154.log 2024-05-04T17:40:23.869 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37309.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-05-04T17:40:23.877 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37393.log 2024-05-04T17:40:23.887 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35154.log: 85.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34354.log 2024-05-04T17:40:23.891 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37393.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35154.log.gz 2024-05-04T17:40:23.895 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36613.log 2024-05-04T17:40:23.898 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34354.log: -- replaced with /var/log/ceph/ceph-client.admin.37393.log.gz 2024-05-04T17:40:23.908 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34354.log.gz 2024-05-04T17:40:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35687.log 2024-05-04T17:40:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36613.log: 90.3% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36613.log.gz 2024-05-04T17:40:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36973.log 2024-05-04T17:40:23.925 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35687.log: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-04T17:40:23.927 INFO:teuthology.orchestra.run.smithi110.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.35687.log.gz 2024-05-04T17:40:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36407.log 2024-05-04T17:40:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36973.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36973.log.gz 2024-05-04T17:40:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2024-05-04T17:40:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36407.log: 87.9% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36407.log.gz 2024-05-04T17:40:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36462.log 2024-05-04T17:40:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34807.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz 2024-05-04T17:40:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34553.log 2024-05-04T17:40:23.934 INFO:teuthology.orchestra.run.smithi136.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-04T17:40:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36343.log 2024-05-04T17:40:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34553.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34553.log.gz 2024-05-04T17:40:23.949 INFO:teuthology.orchestra.run.smithi110.stderr: 95.0% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36462.log.gz 2024-05-04T17:40:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29987.log 2024-05-04T17:40:23.951 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36343.log: 88.0% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36343.log.gz 2024-05-04T17:40:23.952 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36505.log 2024-05-04T17:40:23.952 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29987.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29987.log.gz 2024-05-04T17:40:23.952 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35491.log 2024-05-04T17:40:23.965 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34410.log 2024-05-04T17:40:23.965 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35491.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35491.log.gz 2024-05-04T17:40:23.978 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36931.log 2024-05-04T17:40:23.978 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34410.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.34410.log.gz 2024-05-04T17:40:23.993 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37561.log 2024-05-04T17:40:23.993 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36931.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.36931.log.gz 2024-05-04T17:40:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35487.log 2024-05-04T17:40:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37561.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.37561.log.gz 2024-05-04T17:40:24.013 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-05-04T17:40:24.016 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35487.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35487.log.gz 2024-05-04T17:40:24.026 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30410.log 2024-05-04T17:40:24.027 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34804.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-05-04T17:40:24.035 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35811.log 2024-05-04T17:40:24.048 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30410.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.30410.log.gz 2024-05-04T17:40:24.049 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36365.log 2024-05-04T17:40:24.050 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35811.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35811.log.gz 2024-05-04T17:40:24.058 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37183.log 2024-05-04T17:40:24.069 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36365.log: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35142.log 2024-05-04T17:40:24.074 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37183.log: 86.8% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36365.log.gz 2024-05-04T17:40:24.076 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35147.log 2024-05-04T17:40:24.081 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35142.log: -- replaced with /var/log/ceph/ceph-client.admin.37183.log.gz 2024-05-04T17:40:24.088 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36386.log 2024-05-04T17:40:24.089 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35147.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35147.log.gz 2024-05-04T17:40:24.089 INFO:teuthology.orchestra.run.smithi110.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35142.log.gz 2024-05-04T17:40:24.093 INFO:teuthology.orchestra.run.smithi110.stderr: 89.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-04T17:40:24.094 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34637.log 2024-05-04T17:40:24.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36386.log: 88.1% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36386.log.gz 2024-05-04T17:40:24.096 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-05-04T17:40:24.097 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34637.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34637.log.gz 2024-05-04T17:40:24.097 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.36550.log 2024-05-04T17:40:24.097 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37477.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-05-04T17:40:24.099 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35854.log 2024-05-04T17:40:24.099 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.libcephsqlite.36550.log: 88.1% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36550.log.gz 2024-05-04T17:40:24.100 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35854.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35854.log.gz 2024-05-04T17:40:24.674 INFO:teuthology.orchestra.run.smithi110.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36505.log.gz 2024-05-04T17:40:25.085 INFO:teuthology.orchestra.run.smithi110.stderr: 95.9% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.36573.log.gz 2024-05-04T17:40:27.357 INFO:teuthology.orchestra.run.smithi110.stderr: 95.1% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.35894.log.gz 2024-05-04T17:40:27.828 INFO:teuthology.orchestra.run.smithi110.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-04T17:40:28.501 INFO:teuthology.orchestra.run.smithi136.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-04T17:40:32.718 INFO:teuthology.orchestra.run.smithi110.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-04T17:40:40.251 INFO:teuthology.orchestra.run.smithi136.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-04T17:40:42.051 INFO:teuthology.orchestra.run.smithi110.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-04T17:40:42.939 INFO:teuthology.orchestra.run.smithi136.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-04T17:40:43.044 INFO:teuthology.orchestra.run.smithi110.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-04T17:40:43.046 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-05-04T17:40:43.046 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m19.314s 2024-05-04T17:40:43.046 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m55.016s 2024-05-04T17:40:43.046 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m1.792s 2024-05-04T17:40:43.242 INFO:teuthology.orchestra.run.smithi136.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-04T17:40:43.245 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-05-04T17:40:43.245 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m19.467s 2024-05-04T17:40:43.245 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m57.974s 2024-05-04T17:40:43.245 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m1.876s 2024-05-04T17:40:43.245 INFO:tasks.ceph:Archiving logs... 2024-05-04T17:40:43.245 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/remote/smithi110/log 2024-05-04T17:40:43.246 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-04T17:40:51.901 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/remote/smithi136/log 2024-05-04T17:40:51.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-04T17:41:01.461 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-04T17:41:01.472 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-04T17:41:01.473 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-04T17:41:01.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-04T17:41:01.564 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-04T17:41:01.674 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-04T17:41:01.674 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T17:41:01.677 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T17:41:01.690 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-05-04T17:41:01.693 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-05-04T17:41:01.695 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T17:41:01.695 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-05-04T17:41:01.695 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +827us[ +807us] +/- 52ms 2024-05-04T17:41:01.695 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -934us[ -934us] +/- 60ms 2024-05-04T17:41:01.695 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -98us[ -98us] +/- 92ms 2024-05-04T17:41:01.695 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:41:01.699 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T17:41:01.699 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-05-04T17:41:01.699 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +835us[ +837us] +/- 52ms 2024-05-04T17:41:01.699 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 -913us[ -911us] +/- 60ms 2024-05-04T17:41:01.699 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -86us[ -84us] +/- 92ms 2024-05-04T17:41:01.699 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:41:01.699 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-04T17:41:01.709 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-04T17:41:01.710 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-04T17:41:01.719 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-04T17:41:01.799 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-04T17:41:01.863 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T17:41:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:41:01.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T17:41:01.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:41:01.919 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-04T17:41:01.928 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-04T17:41:01.937 INFO:teuthology.task.internal:Duration was 688.931425 seconds 2024-05-04T17:41:01.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-04T17:41:01.950 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-04T17:41:01.950 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-04T17:41:01.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-04T17:41:01.987 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T17:41:01.997 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T17:41:02.256 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-04T17:41:02.256 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-05-04T17:41:02.256 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-04T17:41:02.295 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-05-04T17:41:02.296 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-04T17:41:02.320 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-04T17:41:02.320 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-04T17:41:02.339 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-04T17:41:02.386 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-04T17:41:02.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-04T17:41:02.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-04T17:41:02.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-04T17:41:02.530 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-04T17:41:02.531 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-04T17:41:02.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-04T17:41:02.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-04T17:41:02.775 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-04T17:41:02.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-04T17:41:02.798 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-05-04T17:41:02.799 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-05-04T17:41:02.821 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-04T17:41:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:41:02.869 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-04T17:41:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:41:02.884 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-04T17:41:02.894 INFO:teuthology.task.internal:Transferring archived files... 2024-05-04T17:41:02.895 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/remote/smithi110 2024-05-04T17:41:02.895 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-04T17:41:02.942 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687130/remote/smithi136 2024-05-04T17:41:02.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-04T17:41:02.974 INFO:teuthology.task.internal:Removing archive directory... 2024-05-04T17:41:02.974 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-04T17:41:02.978 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-04T17:41:03.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-04T17:41:03.080 INFO:teuthology.task.internal:Not uploading archives. 2024-05-04T17:41:03.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-04T17:41:03.090 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-04T17:41:03.090 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-04T17:41:03.092 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-04T17:41:03.104 INFO:teuthology.orchestra.run.smithi110.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 4 17:41 /home/ubuntu/cephtest 2024-05-04T17:41:03.106 INFO:teuthology.orchestra.run.smithi136.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 4 17:41 /home/ubuntu/cephtest 2024-05-04T17:41:03.107 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-04T17:41:03.118 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-04T17:41:03.351 INFO:teuthology.run:Summary data: description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} duration: 688.9314250946045 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-05-04T17:41:03.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-04T17:41:03.442 INFO:teuthology.run:pass