2023-08-22T22:59:22.803 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-22T22:59:22.804 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --owner scheduled_yuriw@teuthology --verbose --name yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793 -- /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/orig.config.yaml 2023-08-22T22:59:22.844 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-22T22:59:22.932 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793 branch: wip-yuri10-testing-2023-08-17-1444 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7376793' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri10-testing-2023-08-17-1444 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: f0ed704638a5702969ce59fb4418946a60474648 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: f0ed704638a5702969ce59fb4418946a60474648 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2023-08-17-1444 sha1: f0ed704638a5702969ce59fb4418946a60474648 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4042 sha1: f0ed704638a5702969ce59fb4418946a60474648 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2023-08-17-1444 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f0ed704638a5702969ce59fb4418946a60474648 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+3T+PSpbH3t890q4p+Mct7FKdX102eb77eXtY6aQVYryOmE5LO4b2qE6gLs6SdX09JTZKaLeLIpqbWMKtb1Vc= smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG7NfRtW1+ZXWRHpP4aldS+1SQUMyv292c+czbs0gFMRfifl6nyklgZeVwlja+SpA4PHEyREN0RAP93I5k6s4eo= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 449a1bc2027504e7b3c3d7b30fa4178906581da7 timestamp: 2023-08-22_18:16:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2224 2023-08-22T22:59:22.932 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa; will attempt to use it 2023-08-22T22:59:22.933 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks 2023-08-22T22:59:22.933 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-22T22:59:22.935 INFO:teuthology.task.internal:Checking packages... 2023-08-22T22:59:22.965 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f0ed704638a5702969ce59fb4418946a60474648' 2023-08-22T22:59:22.965 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-22T22:59:22.965 INFO:teuthology.packaging:ref: None 2023-08-22T22:59:22.966 INFO:teuthology.packaging:tag: None 2023-08-22T22:59:22.966 INFO:teuthology.packaging:branch: wip-yuri10-testing-2023-08-17-1444 2023-08-22T22:59:22.966 INFO:teuthology.packaging:sha1: f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T22:59:22.966 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri10-testing-2023-08-17-1444 2023-08-22T22:59:23.181 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5573.gf0ed7046 2023-08-22T22:59:23.182 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-22T22:59:23.198 INFO:teuthology.task.internal:no buildpackages task found 2023-08-22T22:59:23.199 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-22T22:59:23.207 INFO:teuthology.task.internal:Saving configuration 2023-08-22T22:59:23.232 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-22T22:59:23.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-22T22:59:23.268 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-22 22:54:09.061993', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+3T+PSpbH3t890q4p+Mct7FKdX102eb77eXtY6aQVYryOmE5LO4b2qE6gLs6SdX09JTZKaLeLIpqbWMKtb1Vc='} 2023-08-22T22:59:23.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-22 22:54:09.060576', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG7NfRtW1+ZXWRHpP4aldS+1SQUMyv292c+czbs0gFMRfifl6nyklgZeVwlja+SpA4PHEyREN0RAP93I5k6s4eo='} 2023-08-22T22:59:23.291 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-22T22:59:23.300 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-08-22T22:59:23.301 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-08-22T22:59:23.301 INFO:teuthology.run_tasks:Running task console_log... 2023-08-22T22:59:23.386 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3af54aeee0>, signals=[15]) 2023-08-22T22:59:23.387 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-22T22:59:23.395 INFO:teuthology.task.internal:Opening connections... 2023-08-22T22:59:23.395 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T22:59:23.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T22:59:23.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T22:59:23.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T22:59:23.606 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-08-22T22:59:23.617 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-08-22T22:59:23.635 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-08-22T22:59:23.636 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-08-22T22:59:23.696 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-08-22T22:59:23.696 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2023-08-22T22:59:23.696 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-08-22T22:59:23.697 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-08-22T22:59:23.697 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2023-08-22T22:59:23.697 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2023-08-22T22:59:23.697 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-22T22:59:23.697 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-08-22T22:59:23.698 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2023-08-22T22:59:23.698 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-22T22:59:23.698 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-08-22T22:59:23.698 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-22T22:59:23.698 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-22T22:59:23.698 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-22T22:59:23.699 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-08-22T22:59:23.724 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2023-08-22T22:59:23.744 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2023-08-22T22:59:23.744 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2023-08-22T22:59:23.802 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2023-08-22T22:59:23.802 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2023-08-22T22:59:23.802 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2023-08-22T22:59:23.802 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-22T22:59:23.803 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2023-08-22T22:59:23.804 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-22T22:59:23.804 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-22T22:59:23.804 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-22T22:59:23.804 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2023-08-22T22:59:23.824 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-08-22T22:59:23.832 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-08-22T22:59:23.846 INFO:teuthology.task.internal:Checking for old test directory... 2023-08-22T22:59:23.847 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-08-22T22:59:23.849 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2023-08-22T22:59:23.866 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-08-22T22:59:23.871 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-08-22T22:59:23.872 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-08-22T22:59:23.906 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2023-08-22T22:59:23.933 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-08-22T22:59:23.983 INFO:teuthology.run_tasks:Running task kernel... 2023-08-22T22:59:24.001 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-08-22T22:59:24.002 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-08-22T22:59:24.002 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-22T22:59:24.003 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-22T22:59:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T22:59:24.021 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-08-22T22:59:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T22:59:24.022 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2023-08-22T22:59:24.079 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-239.el9.x86_64 2023-08-22T22:59:24.079 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-239.el9.x86_64 2023-08-22T22:59:24.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2023-08-22T22:59:24.080 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-239.el9.x86_64 2023-08-22T22:59:24.081 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-239.el9.x86_64 2023-08-22T22:59:24.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2023-08-22T22:59:26.061 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 13 MB/s | 6.2 MB 00:00 2023-08-22T22:59:26.911 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 10 MB/s | 6.2 MB 00:00 2023-08-22T22:59:27.545 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2023-08-22T22:59:28.287 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2023-08-22T22:59:29.577 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 7.8 MB 00:01 2023-08-22T22:59:30.197 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.8 MB 00:01 2023-08-22T22:59:31.379 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 24 MB/s | 18 MB 00:00 2023-08-22T22:59:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 14 kB 00:00 2023-08-22T22:59:35.848 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 24 MB/s | 18 MB 00:00 2023-08-22T22:59:41.887 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-08-22T22:59:41.982 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T22:59:41.994 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T22:59:41.994 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T22:59:41.995 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T22:59:41.995 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T22:59:41.995 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-356.el9 CentOS-BaseOS 5.0 M 2023-08-22T22:59:41.995 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T22:59:41.996 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-356.el9 CentOS-BaseOS 20 M 2023-08-22T22:59:41.996 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-356.el9 CentOS-BaseOS 37 M 2023-08-22T22:59:41.996 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-356.el9 CentOS-BaseOS 32 M 2023-08-22T22:59:41.996 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T22:59:41.996 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T22:59:41.997 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T22:59:41.997 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2023-08-22T22:59:41.997 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T22:59:41.997 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 93 M 2023-08-22T22:59:41.997 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 120 M 2023-08-22T22:59:41.998 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T22:59:42.373 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-356.el9.x86_64.rpm 13 MB/s | 5.0 MB 00:00 2023-08-22T22:59:42.941 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-356.el9.x86_64.rpm 21 MB/s | 20 MB 00:00 2023-08-22T22:59:43.757 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-core-5.14.0-356.el9.x86_6 23 MB/s | 32 MB 00:01 2023-08-22T22:59:44.124 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-5.14.0-356.el9.x86_64.rpm 17 MB/s | 37 MB 00:02 2023-08-22T22:59:44.127 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T22:59:44.128 INFO:teuthology.orchestra.run.smithi052.stdout:Total 44 MB/s | 93 MB 00:02 2023-08-22T22:59:44.128 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T22:59:44.532 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T22:59:44.533 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T22:59:45.343 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T22:59:45.343 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T22:59:46.541 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 18 MB 00:15 2023-08-22T22:59:47.638 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T22:59:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-356.el9.x86_64 1/4 2023-08-22T22:59:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-356.el9.x86_64 2/4 2023-08-22T22:59:50.793 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 14 kB 00:00 2023-08-22T22:59:50.799 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-356.el9.x86_64 2/4 2023-08-22T22:59:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-356.el9.x86_64 3/4 2023-08-22T22:59:51.755 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 21 MB/s | 18 MB 00:00 2023-08-22T22:59:55.905 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-356.el9.x86_64 3/4 2023-08-22T22:59:56.027 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-356.el9.x86_64 4/4 2023-08-22T22:59:57.995 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-08-22T22:59:58.111 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T22:59:58.133 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T22:59:58.133 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T22:59:58.134 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T22:59:58.134 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T22:59:58.134 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-356.el9 CentOS-BaseOS 5.0 M 2023-08-22T22:59:58.135 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T22:59:58.135 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-356.el9 CentOS-BaseOS 20 M 2023-08-22T22:59:58.135 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-356.el9 CentOS-BaseOS 37 M 2023-08-22T22:59:58.135 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-356.el9 CentOS-BaseOS 32 M 2023-08-22T22:59:58.136 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T22:59:58.136 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T22:59:58.136 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T22:59:58.136 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2023-08-22T22:59:58.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T22:59:58.137 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 93 M 2023-08-22T22:59:58.137 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 120 M 2023-08-22T22:59:58.137 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T22:59:58.592 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-356.el9.x86_64.rpm 11 MB/s | 5.0 MB 00:00 2023-08-22T22:59:59.125 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-356.el9.x86_64.rpm 20 MB/s | 20 MB 00:00 2023-08-22T23:00:00.064 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-core-5.14.0-356.el9.x86_6 22 MB/s | 32 MB 00:01 2023-08-22T23:00:00.752 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-5.14.0-356.el9.x86_64.rpm 14 MB/s | 37 MB 00:02 2023-08-22T23:00:00.755 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:00:00.755 INFO:teuthology.orchestra.run.smithi029.stdout:Total 35 MB/s | 93 MB 00:02 2023-08-22T23:00:00.755 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:00:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:01.165 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:00:01.165 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:00:01.971 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:00:01.972 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:00:04.291 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:00:05.042 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-356.el9.x86_64 1/4 2023-08-22T23:00:05.309 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-356.el9.x86_64 2/4 2023-08-22T23:00:07.659 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-356.el9.x86_64 2/4 2023-08-22T23:00:07.933 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-356.el9.x86_64 3/4 2023-08-22T23:00:12.863 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-356.el9.x86_64 3/4 2023-08-22T23:00:13.023 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:17.968 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:31.976 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:32.606 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:32.606 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-356.el9.x86_64 1/4 2023-08-22T23:00:32.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-356.el9.x86_64 2/4 2023-08-22T23:00:32.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-356.el9.x86_64 3/4 2023-08-22T23:00:33.083 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:33.084 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:00:33.084 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:00:33.084 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-356.el9.x86_64 2023-08-22T23:00:33.085 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-356.el9.x86_64 2023-08-22T23:00:33.085 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-356.el9.x86_64 2023-08-22T23:00:33.085 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-356.el9.x86_64 2023-08-22T23:00:33.085 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:00:33.085 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:00:33.629 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2023-08-22T23:00:33.721 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-356.el9.x86_64 2023-08-22T23:00:33.721 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-356.el9.x86_64 2023-08-22T23:00:33.722 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-239.el9.x86_64 Expected: 5.14.0-356.el9.x86_64 2023-08-22T23:00:33.722 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-08-22T23:00:33.722 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-08-22T23:00:33.722 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-08-22T23:00:33.722 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2023-08-22T23:00:33.723 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2023-08-22T23:00:33.794 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-356.el9.x86_64 2023-08-22T23:00:33.794 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-356.el9.x86_64 2023-08-22T23:00:33.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2023-08-22T23:00:33.833 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2023-08-22T23:00:33.833 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2023-08-22T23:00:33.833 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 53.el9 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Mon 30 Jan 2023 10:38:40 AM UTC 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8551923 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2023-08-22T23:00:33.834 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Wed 18 Jan 2023 04:40:07 PM UTC, Key ID 05b555b38483c65d 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-53.el9.src.rpm 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Fri 13 Jan 2023 09:07:28 PM UTC 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2023-08-22T23:00:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:00:33.836 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2023-08-22T23:00:33.837 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-08-22T23:00:33.838 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-356.el9.x86_64 2023-08-22T23:00:33.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-08-22T23:00:34.624 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2023-08-22T23:00:36.107 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-08-22T23:00:36.156 INFO:teuthology.orchestra.run.smithi052.stderr:done 2023-08-22T23:00:36.168 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2023-08-22T23:00:36.225 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.VMFRraDKgS 2023-08-22T23:00:36.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VMFRraDKgS 2023-08-22T23:00:36.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.VMFRraDKgS 2023-08-22T23:00:36.410 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.VMFRraDKgS is 6KB 2023-08-22T23:00:36.458 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.VMFRraDKgS 2023-08-22T23:00:36.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2023-08-22T23:00:36.542 INFO:teuthology.orchestra.run.smithi052.stdout:e13cba1d25f14c51a1d04da8db823a4e-0-rescue.conf 2023-08-22T23:00:36.542 INFO:teuthology.orchestra.run.smithi052.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-239.el9.x86_64.conf 2023-08-22T23:00:36.543 INFO:teuthology.orchestra.run.smithi052.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-356.el9.x86_64.conf 2023-08-22T23:00:36.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default e13cba1d25f14c51a1d04da8db823a4e-5.14.0-356.el9.x86_64 2023-08-22T23:00:36.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2023-08-22T23:00:51.087 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:51.129 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:51.948 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-356.el9.x86_64 1/4 2023-08-22T23:00:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-356.el9.x86_64 2/4 2023-08-22T23:00:51.949 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-356.el9.x86_64 3/4 2023-08-22T23:00:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-356.el9.x86_64 4/4 2023-08-22T23:00:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:00:52.440 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:00:52.440 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-356.el9.x86_64 2023-08-22T23:00:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-356.el9.x86_64 2023-08-22T23:00:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-356.el9.x86_64 2023-08-22T23:00:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-356.el9.x86_64 2023-08-22T23:00:52.441 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:00:52.441 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:00:53.233 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-08-22T23:00:53.336 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-356.el9.x86_64 2023-08-22T23:00:53.337 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-356.el9.x86_64 2023-08-22T23:00:53.337 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-239.el9.x86_64 Expected: 5.14.0-356.el9.x86_64 2023-08-22T23:00:53.338 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-08-22T23:00:53.338 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-08-22T23:00:53.338 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-08-22T23:00:53.338 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2023-08-22T23:00:53.338 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-08-22T23:00:53.409 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-356.el9.x86_64 2023-08-22T23:00:53.409 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-356.el9.x86_64 2023-08-22T23:00:53.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-08-22T23:00:53.447 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-08-22T23:00:53.447 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-08-22T23:00:53.447 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 53.el9 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Mon 30 Jan 2023 10:38:40 AM UTC 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8551923 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-08-22T23:00:53.448 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Wed 18 Jan 2023 04:40:07 PM UTC, Key ID 05b555b38483c65d 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-53.el9.src.rpm 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Fri 13 Jan 2023 09:07:28 PM UTC 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-08-22T23:00:53.449 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-08-22T23:00:53.450 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:00:53.451 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-08-22T23:00:53.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-08-22T23:00:53.452 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-356.el9.x86_64 2023-08-22T23:00:53.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-08-22T23:00:54.397 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-08-22T23:00:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-08-22T23:00:56.289 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-08-22T23:00:56.303 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-08-22T23:00:56.327 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.2CF9DXV8cf 2023-08-22T23:00:56.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2CF9DXV8cf 2023-08-22T23:00:56.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.2CF9DXV8cf 2023-08-22T23:00:56.537 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.2CF9DXV8cf is 6KB 2023-08-22T23:00:56.555 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.2CF9DXV8cf 2023-08-22T23:00:56.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-08-22T23:00:56.638 INFO:teuthology.orchestra.run.smithi029.stdout:e13cba1d25f14c51a1d04da8db823a4e-0-rescue.conf 2023-08-22T23:00:56.639 INFO:teuthology.orchestra.run.smithi029.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-239.el9.x86_64.conf 2023-08-22T23:00:56.639 INFO:teuthology.orchestra.run.smithi029.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-356.el9.x86_64.conf 2023-08-22T23:00:56.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default e13cba1d25f14c51a1d04da8db823a4e-5.14.0-356.el9.x86_64 2023-08-22T23:00:56.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-08-22T23:01:06.740 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-08-22T23:01:06.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-08-22T23:01:06.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T23:01:26.836 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-08-22T23:01:26.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-08-22T23:01:26.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T23:02:06.780 DEBUG:teuthology.orchestra.remote:timed out 2023-08-22T23:02:15.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-08-22T23:02:15.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T23:02:16.058 DEBUG:teuthology.orchestra.run.smithi052:> true 2023-08-22T23:02:16.412 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-08-22T23:02:16.412 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-356.el9.x86_64"... 2023-08-22T23:02:16.413 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2023-08-22T23:02:16.427 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-356.el9.x86_64 2023-08-22T23:02:16.427 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-356.el9.x86_64 vs 5.14.0-356.el9.x86_64 2023-08-22T23:02:16.427 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-08-22T23:02:16.427 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-08-22T23:02:17.428 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-08-22T23:02:17.428 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-22T23:02:17.526 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2023-08-22T23:02:17.596 DEBUG:teuthology.parallel:result is None 2023-08-22T23:02:26.838 DEBUG:teuthology.orchestra.remote:timed out 2023-08-22T23:02:35.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-08-22T23:02:35.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T23:02:43.418 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-08-22T23:02:43.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-08-22T23:02:43.903 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-356.el9.x86_64"... 2023-08-22T23:02:43.903 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-08-22T23:02:43.919 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-356.el9.x86_64 2023-08-22T23:02:43.919 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-356.el9.x86_64 vs 5.14.0-356.el9.x86_64 2023-08-22T23:02:43.919 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-08-22T23:02:43.919 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-08-22T23:02:44.920 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-08-22T23:02:44.920 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-22T23:02:45.046 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2023-08-22T23:02:45.083 DEBUG:teuthology.parallel:result is None 2023-08-22T23:02:45.083 INFO:teuthology.run_tasks:Running task internal.base... 2023-08-22T23:02:45.091 INFO:teuthology.task.internal:Creating test directory... 2023-08-22T23:02:45.092 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-22T23:02:45.094 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-22T23:02:45.113 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-08-22T23:02:45.121 INFO:teuthology.run_tasks:Running task internal.archive... 2023-08-22T23:02:45.130 INFO:teuthology.task.internal:Creating archive directory... 2023-08-22T23:02:45.130 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-22T23:02:45.154 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-22T23:02:45.248 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-08-22T23:02:45.257 INFO:teuthology.task.internal:Enabling coredump saving... 2023-08-22T23:02:45.257 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-22T23:02:45.260 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-22T23:02:45.306 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-22T23:02:45.314 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-22T23:02:45.323 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-22T23:02:45.331 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-22T23:02:45.333 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-08-22T23:02:45.345 INFO:teuthology.task.internal:Configuring sudo... 2023-08-22T23:02:45.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-22T23:02:45.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-22T23:02:45.381 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-08-22T23:02:45.391 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-08-22T23:02:45.392 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-22T23:02:45.422 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-22T23:02:45.438 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-22T23:02:45.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-22T23:02:45.579 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-22T23:02:45.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-22T23:02:45.713 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:02:45.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-22T23:02:45.796 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-22T23:02:45.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-22T23:02:45.910 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-22T23:02:45.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-22T23:02:46.002 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:02:46.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-22T23:02:46.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-08-22T23:02:46.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2023-08-22T23:02:46.163 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-22T23:02:46.195 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-22T23:02:46.417 INFO:teuthology.run_tasks:Running task internal.timer... 2023-08-22T23:02:46.425 INFO:teuthology.task.internal:Starting timer... 2023-08-22T23:02:46.425 INFO:teuthology.run_tasks:Running task pcp... 2023-08-22T23:02:46.447 INFO:teuthology.run_tasks:Running task selinux... 2023-08-22T23:02:46.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-08-22T23:02:46.538 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-08-22T23:02:46.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2023-08-22T23:02:46.610 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2023-08-22T23:02:46.611 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-08-22T23:02:46.612 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-08-22T23:02:46.639 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-08-22T23:02:46.639 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2023-08-22T23:02:46.666 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2023-08-22T23:02:46.666 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi052.front.sepia.ceph.com': 'permissive'} 2023-08-22T23:02:46.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-22T23:02:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:02:46.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-22T23:02:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:02:46.748 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-08-22T23:02:46.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-08-22T23:02:46.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2023-08-22T23:02:46.835 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-08-22T23:02:46.845 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-08-22T23:02:46.851 INFO:teuthology.repo_utils:Fetching main from origin 2023-08-22T23:02:46.998 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-08-22T23:02:47.020 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-08-22T23:02:47.021 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-08-22T23:09:49.948 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2023-08-22T23:09:49.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-08-22T23:09:49.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T23:09:50.049 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-08-22T23:09:50.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-08-22T23:09:50.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-08-22T23:09:50.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-22T23:09:50.218 DEBUG:teuthology.orchestra.run.smithi052:> true 2023-08-22T23:09:50.288 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-08-22T23:09:50.288 INFO:teuthology.run_tasks:Running task clock... 2023-08-22T23:09:50.303 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-08-22T23:09:50.304 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-22T23:09:50.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-22T23:09:50.307 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-22T23:09:50.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-22T23:09:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-22T23:09:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-22T23:09:50.364 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-22T23:09:50.385 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-22T23:09:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-08-22T23:09:50.414 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-08-22T23:09:50.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2023-08-22T23:09:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-22T23:09:50.439 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2023-08-22T23:09:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-22T23:09:50.459 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-22T23:09:50.479 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-22T23:09:50.501 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-08-22T23:09:50.505 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-22T23:09:50.505 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-08-22T23:09:50.506 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.506 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.506 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.506 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.527 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2023-08-22T23:09:50.530 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-22T23:09:50.530 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2023-08-22T23:09:50.530 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.530 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.531 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.531 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:09:50.532 INFO:teuthology.run_tasks:Running task install... 2023-08-22T23:09:50.545 DEBUG:teuthology.task.install:project ceph 2023-08-22T23:09:50.547 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648'}} 2023-08-22T23:09:50.547 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648'} 2023-08-22T23:09:50.548 INFO:teuthology.task.install:Using flavor: default 2023-08-22T23:09:50.561 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-08-22T23:09:50.562 INFO:teuthology.task.install:extra packages: [] 2023-08-22T23:09:50.563 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648', 'tag': None, 'wait_for_package': False} 2023-08-22T23:09:50.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T23:09:50.569 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648', 'tag': None, 'wait_for_package': False} 2023-08-22T23:09:50.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T23:09:50.719 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/ 2023-08-22T23:09:50.720 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5573.gf0ed7046 2023-08-22T23:09:50.742 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/ 2023-08-22T23:09:50.742 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5573.gf0ed7046 2023-08-22T23:09:50.887 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-22T23:09:50.888 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:09:50.889 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-22T23:09:50.908 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-08-17-1444/f0ed704638a5702969ce59fb4418946a60474648/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-22T23:09:50.908 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:09:50.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-22T23:09:50.921 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-08-22T23:09:50.922 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f0ed704638a5702969ce59fb4418946a60474648/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-22T23:09:50.941 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-08-22T23:09:50.941 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f0ed704638a5702969ce59fb4418946a60474648/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-22T23:09:50.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-22T23:09:51.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-22T23:09:51.080 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-22T23:09:51.104 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-22T23:09:51.110 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2023-08-22T23:09:51.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2023-08-22T23:09:51.134 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-08-22T23:09:51.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-08-22T23:09:51.357 INFO:teuthology.orchestra.run.smithi052.stdout:72 files removed 2023-08-22T23:09:51.380 INFO:teuthology.orchestra.run.smithi029.stdout:72 files removed 2023-08-22T23:09:51.385 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2023-08-22T23:09:51.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2023-08-22T23:09:52.138 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 172 kB/s | 78 kB 00:00 2023-08-22T23:09:52.201 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 159 kB/s | 78 kB 00:00 2023-08-22T23:09:52.560 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-08-22T23:09:52.597 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 23 kB/s | 9.4 kB 00:00 2023-08-22T23:09:52.974 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2023-08-22T23:09:52.978 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2023-08-22T23:09:53.395 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 16 MB/s | 6.2 MB 00:00 2023-08-22T23:09:53.482 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 13 MB/s | 6.2 MB 00:00 2023-08-22T23:09:54.743 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-08-22T23:09:54.883 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2023-08-22T23:09:56.075 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 7.8 MB 00:01 2023-08-22T23:10:04.077 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 18 MB 00:06 2023-08-22T23:10:05.359 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 789 kB/s | 7.8 MB 00:10 2023-08-22T23:10:08.450 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 18 MB 00:02 2023-08-22T23:10:09.843 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 5.7 MB 00:02 2023-08-22T23:10:11.278 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 14 kB 00:00 2023-08-22T23:10:12.405 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 18 MB/s | 18 MB 00:01 2023-08-22T23:10:13.542 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 5.7 MB 00:01 2023-08-22T23:10:15.725 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 14 kB 00:01 2023-08-22T23:10:17.348 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 12 MB/s | 18 MB 00:01 2023-08-22T23:10:18.732 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:10:18.754 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:10:18.754 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:10:18.754 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:10:18.754 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:10:18.754 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 8.2 M 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 3.3 M 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 3.0 M 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-08-22T23:10:18.755 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 5.2 M 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 19 M 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 24 k 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 666 k 2023-08-22T23:10:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 473 k 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 4.7 M 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-08-22T23:10:18.757 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 45 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 127 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 162 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 320 k 2023-08-22T23:10:18.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 298 k 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 99 k 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout:Install 28 Packages 2023-08-22T23:10:18.759 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2023-08-22T23:10:18.760 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:18.760 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 54 M 2023-08-22T23:10:18.760 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:10:19.805 INFO:teuthology.orchestra.run.smithi052.stdout:(1/30): ceph-base-18.0.0-5573.gf0ed7046.el9.x86 5.8 MB/s | 5.2 MB 00:00 2023-08-22T23:10:19.880 INFO:teuthology.orchestra.run.smithi052.stdout:(2/30): ceph-selinux-18.0.0-5573.gf0ed7046.el9. 321 kB/s | 24 kB 00:00 2023-08-22T23:10:20.055 INFO:teuthology.orchestra.run.smithi052.stdout:(3/30): ceph-radosgw-18.0.0-5573.gf0ed7046.el9. 7.2 MB/s | 8.2 MB 00:01 2023-08-22T23:10:20.105 INFO:teuthology.orchestra.run.smithi052.stdout:(4/30): libcephfs2-18.0.0-5573.gf0ed7046.el9.x8 2.9 MB/s | 666 kB 00:00 2023-08-22T23:10:20.148 INFO:teuthology.orchestra.run.smithi052.stdout:(5/30): libradosstriper1-18.0.0-5573.gf0ed7046. 5.0 MB/s | 473 kB 00:00 2023-08-22T23:10:20.231 INFO:teuthology.orchestra.run.smithi052.stdout:(6/30): python3-ceph-argparse-18.0.0-5573.gf0ed 542 kB/s | 45 kB 00:00 2023-08-22T23:10:20.323 INFO:teuthology.orchestra.run.smithi052.stdout:(7/30): python3-ceph-common-18.0.0-5573.gf0ed70 1.4 MB/s | 127 kB 00:00 2023-08-22T23:10:20.415 INFO:teuthology.orchestra.run.smithi052.stdout:(8/30): python3-cephfs-18.0.0-5573.gf0ed7046.el 1.7 MB/s | 162 kB 00:00 2023-08-22T23:10:20.524 INFO:teuthology.orchestra.run.smithi052.stdout:(9/30): python3-rados-18.0.0-5573.gf0ed7046.el9 2.9 MB/s | 320 kB 00:00 2023-08-22T23:10:20.732 INFO:teuthology.orchestra.run.smithi052.stdout:(10/30): ceph-common-18.0.0-5573.gf0ed7046.el9. 10 MB/s | 19 MB 00:01 2023-08-22T23:10:20.774 INFO:teuthology.orchestra.run.smithi052.stdout:(11/30): python3-rbd-18.0.0-5573.gf0ed7046.el9. 1.2 MB/s | 298 kB 00:00 2023-08-22T23:10:20.808 INFO:teuthology.orchestra.run.smithi052.stdout:(12/30): python3-rgw-18.0.0-5573.gf0ed7046.el9. 1.3 MB/s | 99 kB 00:00 2023-08-22T23:10:20.893 INFO:teuthology.orchestra.run.smithi052.stdout:(13/30): librgw2-18.0.0-5573.gf0ed7046.el9.x86_ 5.9 MB/s | 4.7 MB 00:00 2023-08-22T23:10:21.083 INFO:teuthology.orchestra.run.smithi052.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 343 kB/s | 106 kB 00:00 2023-08-22T23:10:21.117 INFO:teuthology.orchestra.run.smithi052.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 515 kB/s | 159 kB 00:00 2023-08-22T23:10:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 207 kB/s | 44 kB 00:00 2023-08-22T23:10:21.201 INFO:teuthology.orchestra.run.smithi052.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2023-08-22T23:10:21.244 INFO:teuthology.orchestra.run.smithi052.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2023-08-22T23:10:21.285 INFO:teuthology.orchestra.run.smithi052.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 331 kB/s | 41 kB 00:00 2023-08-22T23:10:21.622 INFO:teuthology.orchestra.run.smithi052.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 266 kB/s | 111 kB 00:00 2023-08-22T23:10:21.710 INFO:teuthology.orchestra.run.smithi052.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 267 kB/s | 24 kB 00:00 2023-08-22T23:10:21.752 INFO:teuthology.orchestra.run.smithi052.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 661 kB/s | 309 kB 00:00 2023-08-22T23:10:21.811 INFO:teuthology.orchestra.run.smithi052.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2023-08-22T23:10:21.853 INFO:teuthology.orchestra.run.smithi052.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2023-08-22T23:10:21.995 INFO:teuthology.orchestra.run.smithi052.stdout:(25/30): libarrow-9.0.0-9.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2023-08-22T23:10:22.028 INFO:teuthology.orchestra.run.smithi052.stdout:(26/30): re2-20211101-2.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2023-08-22T23:10:22.079 INFO:teuthology.orchestra.run.smithi052.stdout:(27/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.1 MB/s | 837 kB 00:00 2023-08-22T23:10:22.137 INFO:teuthology.orchestra.run.smithi052.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-08-22T23:10:22.396 INFO:teuthology.orchestra.run.smithi052.stdout:(29/30): librados2-18.0.0-5573.gf0ed7046.el9.x8 8.9 MB/s | 3.3 MB 00:00 2023-08-22T23:10:22.471 INFO:teuthology.orchestra.run.smithi052.stdout:(30/30): librbd1-18.0.0-5573.gf0ed7046.el9.x86_ 7.7 MB/s | 3.0 MB 00:00 2023-08-22T23:10:22.479 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:10:22.479 INFO:teuthology.orchestra.run.smithi052.stdout:Total 15 MB/s | 54 MB 00:03 2023-08-22T23:10:22.480 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:10:22.521 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:10:22.522 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:10:23.006 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:10:23.007 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:10:23.904 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:10:23.933 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:10:23.933 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:10:23.933 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:10:23.933 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:10:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 8.2 M 2023-08-22T23:10:23.934 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-08-22T23:10:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 3.3 M 2023-08-22T23:10:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 3.0 M 2023-08-22T23:10:23.934 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:10:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options 2023-08-22T23:10:23.935 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-08-22T23:10:23.935 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 5.2 M 2023-08-22T23:10:23.935 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 19 M 2023-08-22T23:10:23.936 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 24 k 2023-08-22T23:10:23.936 INFO:teuthology.orchestra.run.smithi029.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-08-22T23:10:23.936 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-08-22T23:10:23.936 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-08-22T23:10:23.937 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-08-22T23:10:23.937 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 666 k 2023-08-22T23:10:23.937 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-08-22T23:10:23.938 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-08-22T23:10:23.938 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-08-22T23:10:23.938 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 473 k 2023-08-22T23:10:23.938 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-08-22T23:10:23.939 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 4.7 M 2023-08-22T23:10:23.939 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-08-22T23:10:23.939 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-08-22T23:10:23.939 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-08-22T23:10:23.940 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse 2023-08-22T23:10:23.940 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 45 k 2023-08-22T23:10:23.940 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 127 k 2023-08-22T23:10:23.940 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 162 k 2023-08-22T23:10:23.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-08-22T23:10:23.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 320 k 2023-08-22T23:10:23.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 298 k 2023-08-22T23:10:23.941 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 99 k 2023-08-22T23:10:23.942 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-08-22T23:10:23.942 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-08-22T23:10:23.942 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:23.943 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:10:23.943 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:10:23.943 INFO:teuthology.orchestra.run.smithi029.stdout:Install 28 Packages 2023-08-22T23:10:23.943 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2023-08-22T23:10:23.944 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:23.945 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 54 M 2023-08-22T23:10:23.945 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:10:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:10:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-08-22T23:10:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2023-08-22T23:10:24.612 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2023-08-22T23:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout:(1/30): ceph-base-18.0.0-5573.gf0ed7046.el9.x86 6.4 MB/s | 5.2 MB 00:00 2023-08-22T23:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout:(2/30): ceph-selinux-18.0.0-5573.gf0ed7046.el9. 283 kB/s | 24 kB 00:00 2023-08-22T23:10:25.022 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-08-22T23:10:25.075 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/32 2023-08-22T23:10:25.085 INFO:teuthology.orchestra.run.smithi029.stdout:(3/30): ceph-radosgw-18.0.0-5573.gf0ed7046.el9. 7.5 MB/s | 8.2 MB 00:01 2023-08-22T23:10:25.137 INFO:teuthology.orchestra.run.smithi029.stdout:(4/30): libcephfs2-18.0.0-5573.gf0ed7046.el9.x8 2.8 MB/s | 666 kB 00:00 2023-08-22T23:10:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/32 2023-08-22T23:10:25.185 INFO:teuthology.orchestra.run.smithi029.stdout:(5/30): libradosstriper1-18.0.0-5573.gf0ed7046. 4.6 MB/s | 473 kB 00:00 2023-08-22T23:10:25.252 INFO:teuthology.orchestra.run.smithi029.stdout:(6/30): python3-ceph-argparse-18.0.0-5573.gf0ed 674 kB/s | 45 kB 00:00 2023-08-22T23:10:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 6/32 2023-08-22T23:10:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 7/32 2023-08-22T23:10:25.361 INFO:teuthology.orchestra.run.smithi029.stdout:(7/30): python3-ceph-common-18.0.0-5573.gf0ed70 1.1 MB/s | 127 kB 00:00 2023-08-22T23:10:25.383 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 7/32 2023-08-22T23:10:25.426 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-08-22T23:10:25.478 INFO:teuthology.orchestra.run.smithi029.stdout:(8/30): python3-cephfs-18.0.0-5573.gf0ed7046.el 1.4 MB/s | 162 kB 00:00 2023-08-22T23:10:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:18.0.0-5573.gf0ed7046.el9. 9/32 2023-08-22T23:10:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 10/32 2023-08-22T23:10:25.596 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_6 11/32 2023-08-22T23:10:25.615 INFO:teuthology.orchestra.run.smithi029.stdout:(9/30): librgw2-18.0.0-5573.gf0ed7046.el9.x86_6 9.8 MB/s | 4.7 MB 00:00 2023-08-22T23:10:25.656 INFO:teuthology.orchestra.run.smithi029.stdout:(10/30): python3-rados-18.0.0-5573.gf0ed7046.el 1.8 MB/s | 320 kB 00:00 2023-08-22T23:10:25.661 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_6 11/32 2023-08-22T23:10:25.705 INFO:teuthology.orchestra.run.smithi029.stdout:(11/30): python3-rbd-18.0.0-5573.gf0ed7046.el9. 3.2 MB/s | 298 kB 00:00 2023-08-22T23:10:25.756 INFO:teuthology.orchestra.run.smithi029.stdout:(12/30): python3-rgw-18.0.0-5573.gf0ed7046.el9. 986 kB/s | 99 kB 00:00 2023-08-22T23:10:25.792 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-2.el9.x86_64 12/32 2023-08-22T23:10:25.837 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-08-22T23:10:25.894 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-08-22T23:10:25.960 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-08-22T23:10:25.994 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-08-22T23:10:26.031 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-08-22T23:10:26.072 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-08-22T23:10:26.148 INFO:teuthology.orchestra.run.smithi029.stdout:(13/30): ceph-common-18.0.0-5573.gf0ed7046.el9. 8.7 MB/s | 19 MB 00:02 2023-08-22T23:10:26.198 INFO:teuthology.orchestra.run.smithi029.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 215 kB/s | 106 kB 00:00 2023-08-22T23:10:26.224 INFO:teuthology.orchestra.run.smithi029.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 44 kB 00:00 2023-08-22T23:10:26.292 INFO:teuthology.orchestra.run.smithi029.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 297 kB/s | 159 kB 00:00 2023-08-22T23:10:26.344 INFO:teuthology.orchestra.run.smithi029.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 661 kB 00:00 2023-08-22T23:10:26.386 INFO:teuthology.orchestra.run.smithi029.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2023-08-22T23:10:26.396 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-08-22T23:10:26.411 INFO:teuthology.orchestra.run.smithi029.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 617 kB/s | 41 kB 00:00 2023-08-22T23:10:26.469 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 20/32 2023-08-22T23:10:26.558 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 20/32 2023-08-22T23:10:26.599 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 21/32 2023-08-22T23:10:26.659 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-08-22T23:10:27.029 INFO:teuthology.orchestra.run.smithi029.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 162 kB/s | 111 kB 00:00 2023-08-22T23:10:27.079 INFO:teuthology.orchestra.run.smithi029.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 445 kB/s | 309 kB 00:00 2023-08-22T23:10:27.128 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-08-22T23:10:27.130 INFO:teuthology.orchestra.run.smithi029.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 242 kB/s | 24 kB 00:00 2023-08-22T23:10:27.170 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 24/32 2023-08-22T23:10:27.172 INFO:teuthology.orchestra.run.smithi029.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 526 kB/s | 49 kB 00:00 2023-08-22T23:10:27.242 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 24/32 2023-08-22T23:10:27.256 INFO:teuthology.orchestra.run.smithi029.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 535 kB/s | 67 kB 00:00 2023-08-22T23:10:27.329 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 25/32 2023-08-22T23:10:27.383 INFO:teuthology.orchestra.run.smithi029.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2023-08-22T23:10:27.384 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:18.0.0-5573.gf0ed7046.el9.x8 26/32 2023-08-22T23:10:27.507 INFO:teuthology.orchestra.run.smithi029.stdout:(26/30): libarrow-9.0.0-9.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2023-08-22T23:10:27.549 INFO:teuthology.orchestra.run.smithi029.stdout:(27/30): re2-20211101-2.el9.x86_64.rpm 657 kB/s | 192 kB 00:00 2023-08-22T23:10:27.642 INFO:teuthology.orchestra.run.smithi029.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2023-08-22T23:10:27.934 INFO:teuthology.orchestra.run.smithi029.stdout:(29/30): librados2-18.0.0-5573.gf0ed7046.el9.x8 7.7 MB/s | 3.3 MB 00:00 2023-08-22T23:10:28.034 INFO:teuthology.orchestra.run.smithi029.stdout:(30/30): librbd1-18.0.0-5573.gf0ed7046.el9.x86_ 6.3 MB/s | 3.0 MB 00:00 2023-08-22T23:10:28.038 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:10:28.039 INFO:teuthology.orchestra.run.smithi029.stdout:Total 13 MB/s | 54 MB 00:04 2023-08-22T23:10:28.039 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:10:28.081 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:10:28.081 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:10:28.555 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:10:28.555 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:10:29.574 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/32 2023-08-22T23:10:29.610 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/32 2023-08-22T23:10:30.113 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:10:30.208 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/32 2023-08-22T23:10:30.255 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/32 2023-08-22T23:10:30.257 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-08-22T23:10:30.323 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/32 2023-08-22T23:10:30.324 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-08-22T23:10:30.325 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-08-22T23:10:30.325 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:30.327 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2023-08-22T23:10:30.373 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/32 2023-08-22T23:10:30.413 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2023-08-22T23:10:30.771 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-08-22T23:10:30.928 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/32 2023-08-22T23:10:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/32 2023-08-22T23:10:31.155 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 6/32 2023-08-22T23:10:31.207 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 7/32 2023-08-22T23:10:31.304 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 7/32 2023-08-22T23:10:31.366 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-08-22T23:10:31.436 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:18.0.0-5573.gf0ed7046.el9. 9/32 2023-08-22T23:10:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 10/32 2023-08-22T23:10:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_6 11/32 2023-08-22T23:10:31.638 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_6 11/32 2023-08-22T23:10:31.774 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-2.el9.x86_64 12/32 2023-08-22T23:10:31.833 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-08-22T23:10:31.892 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-08-22T23:10:31.955 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-08-22T23:10:32.007 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-08-22T23:10:32.061 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-08-22T23:10:32.118 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-08-22T23:10:32.470 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-08-22T23:10:32.517 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 20/32 2023-08-22T23:10:32.642 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 20/32 2023-08-22T23:10:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 21/32 2023-08-22T23:10:32.761 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-08-22T23:10:33.228 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-08-22T23:10:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 24/32 2023-08-22T23:10:33.379 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 24/32 2023-08-22T23:10:33.475 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 25/32 2023-08-22T23:10:33.528 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:18.0.0-5573.gf0ed7046.el9.x8 26/32 2023-08-22T23:10:35.729 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/32 2023-08-22T23:10:35.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/32 2023-08-22T23:10:36.367 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/32 2023-08-22T23:10:36.448 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/32 2023-08-22T23:10:36.542 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/32 2023-08-22T23:10:36.542 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-08-22T23:10:36.543 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-08-22T23:10:36.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:36.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/32 2023-08-22T23:10:40.771 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/32 2023-08-22T23:10:40.772 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2023-08-22T23:10:40.772 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2023-08-22T23:10:40.772 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2023-08-22T23:10:40.772 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2023-08-22T23:10:40.772 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2023-08-22T23:10:40.773 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2023-08-22T23:10:40.773 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2023-08-22T23:10:40.773 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:41.614 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/32 2023-08-22T23:10:41.641 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/32 2023-08-22T23:10:41.641 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:10:41.641 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-08-22T23:10:41.642 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-22T23:10:41.642 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-22T23:10:41.642 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:41.643 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-08-22T23:10:41.750 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-08-22T23:10:41.750 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-08-22T23:10:43.065 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-08-22T23:10:43.066 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/32 2023-08-22T23:10:43.066 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/32 2023-08-22T23:10:43.066 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 3/32 2023-08-22T23:10:43.066 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 4/32 2023-08-22T23:10:43.066 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/32 2023-08-22T23:10:43.067 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_6 6/32 2023-08-22T23:10:43.067 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 7/32 2023-08-22T23:10:43.067 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5573.gf0ed7046.el9. 8/32 2023-08-22T23:10:43.067 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:18.0.0-5573.gf0ed7046.el9.x8 9/32 2023-08-22T23:10:43.068 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 10/32 2023-08-22T23:10:43.068 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 11/32 2023-08-22T23:10:43.068 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 12/32 2023-08-22T23:10:43.069 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 13/32 2023-08-22T23:10:43.069 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-08-22T23:10:43.069 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-08-22T23:10:43.070 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2023-08-22T23:10:43.071 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2023-08-22T23:10:43.071 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2023-08-22T23:10:43.071 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2023-08-22T23:10:43.072 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-08-22T23:10:43.072 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-08-22T23:10:43.072 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-08-22T23:10:43.073 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-08-22T23:10:43.073 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-08-22T23:10:43.073 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-08-22T23:10:43.073 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-08-22T23:10:43.074 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-2.el9.x86_64 27/32 2023-08-22T23:10:43.074 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-08-22T23:10:43.075 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/32 2023-08-22T23:10:43.075 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-08-22T23:10:43.075 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 31/32 2023-08-22T23:10:43.835 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-08-22T23:10:43.835 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:43.835 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.836 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: fmt-8.1.1-5.el9.x86_64 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-9.el9.x86_64 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.837 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2023-08-22T23:10:43.838 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-08-22T23:10:43.839 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-08-22T23:10:43.839 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.839 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.839 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.839 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-08-22T23:10:43.839 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.840 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.840 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:43.840 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-2.el9.x86_64 2023-08-22T23:10:43.840 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2023-08-22T23:10:43.840 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:43.840 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:10:44.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2023-08-22T23:10:44.779 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:33 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:10:44.916 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:10:44.917 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:10:44.917 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:10:44.917 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:10:44.918 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:10:44.918 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 40 M 2023-08-22T23:10:44.918 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:10:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.6-15.el9 CentOS-AppStream 186 k 2023-08-22T23:10:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 164 k 2023-08-22T23:10:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-08-22T23:10:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-08-22T23:10:44.920 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-08-22T23:10:44.920 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:44.920 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:10:44.920 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:10:44.921 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2023-08-22T23:10:44.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:44.923 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 41 M 2023-08-22T23:10:44.923 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 173 M 2023-08-22T23:10:44.923 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:10:45.139 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): jq-1.6-15.el9.x86_64.rpm 845 kB/s | 186 kB 00:00 2023-08-22T23:10:45.214 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2023-08-22T23:10:45.264 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-08-22T23:10:45.306 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-08-22T23:10:45.407 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): libcephsqlite-18.0.0-5573.gf0ed7046.el9. 336 kB/s | 164 kB 00:00 2023-08-22T23:10:47.299 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-18.0.0-5573.gf0ed7046.el9.x86_ 17 MB/s | 40 MB 00:02 2023-08-22T23:10:47.302 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:10:47.302 INFO:teuthology.orchestra.run.smithi052.stdout:Total 17 MB/s | 41 MB 00:02 2023-08-22T23:10:47.302 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:10:47.325 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:10:47.326 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:10:47.702 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:10:47.702 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:10:48.251 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:10:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-08-22T23:10:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-08-22T23:10:48.475 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-08-22T23:10:48.543 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-08-22T23:10:48.586 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/6 2023-08-22T23:10:48.839 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/32 2023-08-22T23:10:48.839 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-08-22T23:10:48.840 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-08-22T23:10:48.840 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-08-22T23:10:48.841 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-08-22T23:10:48.841 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-08-22T23:10:48.841 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2023-08-22T23:10:48.841 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-08-22T23:10:48.843 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:49.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/32 2023-08-22T23:10:49.706 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/32 2023-08-22T23:10:49.706 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:10:49.706 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-08-22T23:10:49.707 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-22T23:10:49.707 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-22T23:10:49.707 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:49.709 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-08-22T23:10:49.793 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-08-22T23:10:49.793 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-08-22T23:10:51.390 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-08-22T23:10:51.390 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/32 2023-08-22T23:10:51.391 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/32 2023-08-22T23:10:51.391 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 3/32 2023-08-22T23:10:51.391 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 4/32 2023-08-22T23:10:51.391 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/32 2023-08-22T23:10:51.392 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_6 6/32 2023-08-22T23:10:51.392 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 7/32 2023-08-22T23:10:51.392 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5573.gf0ed7046.el9. 8/32 2023-08-22T23:10:51.393 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:18.0.0-5573.gf0ed7046.el9.x8 9/32 2023-08-22T23:10:51.393 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 10/32 2023-08-22T23:10:51.393 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 11/32 2023-08-22T23:10:51.394 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 12/32 2023-08-22T23:10:51.394 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 13/32 2023-08-22T23:10:51.394 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-08-22T23:10:51.394 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-08-22T23:10:51.395 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2023-08-22T23:10:51.395 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2023-08-22T23:10:51.396 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2023-08-22T23:10:51.396 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2023-08-22T23:10:51.396 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-08-22T23:10:51.396 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-08-22T23:10:51.396 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-08-22T23:10:51.396 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-2.el9.x86_64 27/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-08-22T23:10:51.397 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 31/32 2023-08-22T23:10:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-08-22T23:10:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-08-22T23:10:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.309 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:10:52.309 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-08-22T23:10:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.310 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: fmt-8.1.1-5.el9.x86_64 2023-08-22T23:10:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-08-22T23:10:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-9.el9.x86_64 2023-08-22T23:10:52.311 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-08-22T23:10:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2023-08-22T23:10:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-08-22T23:10:52.312 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-08-22T23:10:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-08-22T23:10:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.313 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2023-08-22T23:10:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-08-22T23:10:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-08-22T23:10:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.314 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-08-22T23:10:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-2.el9.x86_64 2023-08-22T23:10:52.316 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2023-08-22T23:10:52.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:52.317 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:10:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/6 2023-08-22T23:10:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 6/6 2023-08-22T23:10:52.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2023-08-22T23:10:53.278 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:37 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:10:53.417 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:10:53.417 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:10:53.417 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 40 M 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-15.el9 CentOS-AppStream 186 k 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 164 k 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-08-22T23:10:53.418 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-08-22T23:10:53.419 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-08-22T23:10:53.419 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:53.419 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:10:53.419 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:10:53.419 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-08-22T23:10:53.419 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:10:53.420 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 41 M 2023-08-22T23:10:53.421 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 173 M 2023-08-22T23:10:53.421 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:10:53.606 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): jq-1.6-15.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-08-22T23:10:53.691 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2023-08-22T23:10:53.775 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2023-08-22T23:10:53.835 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2023-08-22T23:10:53.920 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): libcephsqlite-18.0.0-5573.gf0ed7046.el9. 328 kB/s | 164 kB 00:00 2023-08-22T23:10:55.042 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 6/6 2023-08-22T23:10:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/6 2023-08-22T23:10:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/6 2023-08-22T23:10:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.6-15.el9.x86_64 3/6 2023-08-22T23:10:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-08-22T23:10:55.044 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-08-22T23:10:55.443 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-08-22T23:10:55.443 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:55.443 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:10:55.444 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:55.444 INFO:teuthology.orchestra.run.smithi052.stdout: jq-1.6-15.el9.x86_64 2023-08-22T23:10:55.444 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:10:55.444 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-08-22T23:10:55.444 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2023-08-22T23:10:55.444 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-08-22T23:10:55.445 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:55.445 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:10:55.801 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2023-08-22T23:10:56.037 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-18.0.0-5573.gf0ed7046.el9.x86_ 15 MB/s | 40 MB 00:02 2023-08-22T23:10:56.042 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:10:56.043 INFO:teuthology.orchestra.run.smithi029.stdout:Total 16 MB/s | 41 MB 00:02 2023-08-22T23:10:56.043 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:10:56.066 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:10:56.067 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:10:56.333 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:45 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:10:56.433 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:10:56.434 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:10:56.472 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:10:56.475 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2023-08-22T23:10:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:10:56.475 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2023-08-22T23:10:56.475 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:10:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 6.5 k 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 2.1 M 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.5 M 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 242 k 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 4.4 M 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 18 M 2023-08-22T23:10:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-08-22T23:10:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-08-22T23:10:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-08-22T23:10:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-08-22T23:10:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-08-22T23:10:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-08-22T23:10:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-08-22T23:10:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-08-22T23:10:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-08-22T23:10:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-08-22T23:10:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-08-22T23:10:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-08-22T23:10:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-08-22T23:10:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-08-22T23:10:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-08-22T23:10:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-08-22T23:10:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-08-22T23:10:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-08-22T23:10:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-08-22T23:10:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:56.483 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:10:56.483 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2023-08-22T23:10:56.483 INFO:teuthology.orchestra.run.smithi052.stdout:Install 31 Packages 2023-08-22T23:10:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:10:56.484 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 29 M 2023-08-22T23:10:56.484 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 109 M 2023-08-22T23:10:56.484 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:10:56.780 INFO:teuthology.orchestra.run.smithi052.stdout:(1/31): ceph-18.0.0-5573.gf0ed7046.el9.x86_64.r 31 kB/s | 6.5 kB 00:00 2023-08-22T23:10:56.983 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:10:57.114 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-08-22T23:10:57.161 INFO:teuthology.orchestra.run.smithi052.stdout:(2/31): ceph-mgr-18.0.0-5573.gf0ed7046.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2023-08-22T23:10:57.173 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-08-22T23:10:57.231 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-08-22T23:10:57.280 INFO:teuthology.orchestra.run.smithi052.stdout:(3/31): ceph-mon-18.0.0-5573.gf0ed7046.el9.x86_ 8.7 MB/s | 4.4 MB 00:00 2023-08-22T23:10:57.295 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-08-22T23:10:57.337 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/6 2023-08-22T23:10:57.362 INFO:teuthology.orchestra.run.smithi052.stdout:(4/31): ceph-mgr-modules-core-18.0.0-5573.gf0ed 2.8 MB/s | 242 kB 00:00 2023-08-22T23:10:57.471 INFO:teuthology.orchestra.run.smithi052.stdout:(5/31): ceph-mds-18.0.0-5573.gf0ed7046.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2023-08-22T23:10:57.522 INFO:teuthology.orchestra.run.smithi052.stdout:(6/31): python3-mako-1.1.4-6.el9.noarch.rpm 911 kB/s | 144 kB 00:00 2023-08-22T23:10:57.538 INFO:teuthology.orchestra.run.smithi052.stdout:(7/31): python3-markupsafe-1.1.1-12.el9.x86_64. 473 kB/s | 32 kB 00:00 2023-08-22T23:10:57.555 INFO:teuthology.orchestra.run.smithi052.stdout:(8/31): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-08-22T23:10:57.572 INFO:teuthology.orchestra.run.smithi052.stdout:(9/31): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-08-22T23:10:57.614 INFO:teuthology.orchestra.run.smithi052.stdout:(10/31): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-08-22T23:10:57.673 INFO:teuthology.orchestra.run.smithi052.stdout:(11/31): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2023-08-22T23:10:57.707 INFO:teuthology.orchestra.run.smithi052.stdout:(12/31): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2023-08-22T23:10:57.740 INFO:teuthology.orchestra.run.smithi052.stdout:(13/31): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2023-08-22T23:10:57.898 INFO:teuthology.orchestra.run.smithi052.stdout:(14/31): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 227 kB/s | 43 kB 00:00 2023-08-22T23:10:57.982 INFO:teuthology.orchestra.run.smithi052.stdout:(15/31): python3-cheroot-8.6.0-4.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2023-08-22T23:10:58.049 INFO:teuthology.orchestra.run.smithi052.stdout:(16/31): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2023-08-22T23:10:58.094 INFO:teuthology.orchestra.run.smithi052.stdout:(17/31): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2023-08-22T23:10:58.116 INFO:teuthology.orchestra.run.smithi052.stdout:(18/31): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2023-08-22T23:10:58.142 INFO:teuthology.orchestra.run.smithi052.stdout:(19/31): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2023-08-22T23:10:58.342 INFO:teuthology.orchestra.run.smithi052.stdout:(20/31): ceph-osd-18.0.0-5573.gf0ed7046.el9.x86 15 MB/s | 18 MB 00:01 2023-08-22T23:10:58.367 INFO:teuthology.orchestra.run.smithi052.stdout:(21/31): python3-jaraco-functools-3.5.0-2.el9.n 77 kB/s | 19 kB 00:00 2023-08-22T23:10:58.393 INFO:teuthology.orchestra.run.smithi052.stdout:(22/31): python3-jaraco-text-3.2.0-6.el9.noarch 78 kB/s | 20 kB 00:00 2023-08-22T23:10:58.484 INFO:teuthology.orchestra.run.smithi052.stdout:(23/31): python3-more-itertools-8.12.0-2.el9.no 862 kB/s | 79 kB 00:00 2023-08-22T23:10:58.518 INFO:teuthology.orchestra.run.smithi052.stdout:(24/31): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-08-22T23:10:58.543 INFO:teuthology.orchestra.run.smithi052.stdout:(25/31): python3-logutils-0.3.5-21.el9.noarch.r 230 kB/s | 46 kB 00:00 2023-08-22T23:10:58.569 INFO:teuthology.orchestra.run.smithi052.stdout:(26/31): python3-portend-3.1.0-2.el9.noarch.rpm 195 kB/s | 16 kB 00:00 2023-08-22T23:10:58.605 INFO:teuthology.orchestra.run.smithi052.stdout:(27/31): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2023-08-22T23:10:58.628 INFO:teuthology.orchestra.run.smithi052.stdout:(28/31): python3-tempora-5.0.0-2.el9.noarch.rpm 424 kB/s | 36 kB 00:00 2023-08-22T23:10:58.670 INFO:teuthology.orchestra.run.smithi052.stdout:(29/31): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-08-22T23:10:58.703 INFO:teuthology.orchestra.run.smithi052.stdout:(30/31): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2023-08-22T23:10:58.728 INFO:teuthology.orchestra.run.smithi052.stdout:(31/31): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2023-08-22T23:10:58.732 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:10:58.733 INFO:teuthology.orchestra.run.smithi052.stdout:Total 13 MB/s | 29 MB 00:02 2023-08-22T23:10:58.733 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:10:58.773 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:10:58.773 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:10:59.120 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:10:59.120 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:00.356 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/31 2023-08-22T23:11:00.397 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/31 2023-08-22T23:11:00.438 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/31 2023-08-22T23:11:00.543 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/31 2023-08-22T23:11:00.634 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/31 2023-08-22T23:11:00.683 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/31 2023-08-22T23:11:00.737 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/31 2023-08-22T23:11:00.811 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 8/31 2023-08-22T23:11:00.875 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 9/31 2023-08-22T23:11:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 10/31 2023-08-22T23:11:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 11/31 2023-08-22T23:11:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 12/31 2023-08-22T23:11:01.257 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 13/31 2023-08-22T23:11:01.303 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/31 2023-08-22T23:11:01.391 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/6 2023-08-22T23:11:01.398 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/31 2023-08-22T23:11:01.463 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-08-22T23:11:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 6/6 2023-08-22T23:11:01.505 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 17/31 2023-08-22T23:11:01.554 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 18/31 2023-08-22T23:11:01.606 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/31 2023-08-22T23:11:01.648 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 20/31 2023-08-22T23:11:01.757 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 21/31 2023-08-22T23:11:01.821 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 22/31 2023-08-22T23:11:01.912 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/31 2023-08-22T23:11:02.062 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 24/31 2023-08-22T23:11:02.205 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-08-22T23:11:02.375 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5573.gf0ed7046.el9. 26/31 2023-08-22T23:11:02.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/31 2023-08-22T23:11:02.444 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/31 2023-08-22T23:11:02.444 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:02.445 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-08-22T23:11:02.445 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-22T23:11:02.445 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-22T23:11:02.445 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:04.069 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/31 2023-08-22T23:11:04.096 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/31 2023-08-22T23:11:04.097 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:04.097 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-08-22T23:11:04.097 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-22T23:11:04.097 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-22T23:11:04.097 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:04.578 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/31 2023-08-22T23:11:04.603 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/31 2023-08-22T23:11:04.603 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:04.603 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-08-22T23:11:04.604 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-22T23:11:04.604 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-22T23:11:04.604 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:04.820 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/31 2023-08-22T23:11:04.845 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/31 2023-08-22T23:11:04.845 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:04.845 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-08-22T23:11:04.845 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-22T23:11:04.845 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-22T23:11:04.846 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:04.919 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 31/31 2023-08-22T23:11:05.505 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 6/6 2023-08-22T23:11:05.505 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/6 2023-08-22T23:11:05.505 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/6 2023-08-22T23:11:05.506 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-15.el9.x86_64 3/6 2023-08-22T23:11:05.506 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-08-22T23:11:05.506 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-08-22T23:11:05.926 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-15.el9.x86_64 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-08-22T23:11:05.927 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2023-08-22T23:11:05.928 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-08-22T23:11:05.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:06.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2023-08-22T23:11:06.866 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 31/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 3/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 4/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5573.gf0ed7046.el9. 6/31 2023-08-22T23:11:06.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/31 2023-08-22T23:11:06.868 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/31 2023-08-22T23:11:06.868 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/31 2023-08-22T23:11:06.869 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/31 2023-08-22T23:11:06.869 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/31 2023-08-22T23:11:06.869 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/31 2023-08-22T23:11:06.870 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/31 2023-08-22T23:11:06.870 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/31 2023-08-22T23:11:06.870 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 15/31 2023-08-22T23:11:06.871 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-08-22T23:11:06.871 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 17/31 2023-08-22T23:11:06.871 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 18/31 2023-08-22T23:11:06.871 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 19/31 2023-08-22T23:11:06.872 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 20/31 2023-08-22T23:11:06.872 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 21/31 2023-08-22T23:11:06.872 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 22/31 2023-08-22T23:11:06.872 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 23/31 2023-08-22T23:11:06.873 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 24/31 2023-08-22T23:11:06.873 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-08-22T23:11:06.873 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 26/31 2023-08-22T23:11:06.874 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/31 2023-08-22T23:11:06.875 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 28/31 2023-08-22T23:11:06.875 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 29/31 2023-08-22T23:11:06.875 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 30/31 2023-08-22T23:11:06.894 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:50 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:07.037 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:07.038 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2023-08-22T23:11:07.038 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:07.038 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2023-08-22T23:11:07.038 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:07.038 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 6.5 k 2023-08-22T23:11:07.038 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 2.1 M 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.5 M 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 242 k 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 4.4 M 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 18 M 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-08-22T23:11:07.039 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-08-22T23:11:07.040 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-08-22T23:11:07.041 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-08-22T23:11:07.041 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-08-22T23:11:07.041 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-08-22T23:11:07.041 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-08-22T23:11:07.042 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:07.043 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2023-08-22T23:11:07.044 INFO:teuthology.orchestra.run.smithi029.stdout:Install 31 Packages 2023-08-22T23:11:07.044 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:07.045 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 29 M 2023-08-22T23:11:07.045 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 109 M 2023-08-22T23:11:07.046 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:07.324 INFO:teuthology.orchestra.run.smithi029.stdout:(1/31): ceph-18.0.0-5573.gf0ed7046.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2023-08-22T23:11:07.467 INFO:teuthology.orchestra.run.smithi029.stdout:(2/31): ceph-mgr-18.0.0-5573.gf0ed7046.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2023-08-22T23:11:07.785 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 31/31 2023-08-22T23:11:07.785 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:07.786 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:07.786 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:07.786 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:07.786 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:07.787 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:07.787 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:07.787 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:07.788 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-08-22T23:11:07.788 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-08-22T23:11:07.788 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-08-22T23:11:07.789 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-08-22T23:11:07.789 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-08-22T23:11:07.789 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-08-22T23:11:07.789 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-08-22T23:11:07.789 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-08-22T23:11:07.790 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-08-22T23:11:07.790 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-08-22T23:11:07.790 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-08-22T23:11:07.791 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2023-08-22T23:11:07.791 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-08-22T23:11:07.791 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-08-22T23:11:07.792 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2023-08-22T23:11:07.793 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2023-08-22T23:11:07.794 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-08-22T23:11:07.794 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-08-22T23:11:07.794 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:07.794 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:08.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2023-08-22T23:11:08.259 INFO:teuthology.orchestra.run.smithi029.stdout:(3/31): ceph-mds-18.0.0-5573.gf0ed7046.el9.x86_ 1.8 MB/s | 2.1 MB 00:01 2023-08-22T23:11:08.585 INFO:teuthology.orchestra.run.smithi029.stdout:(4/31): ceph-osd-18.0.0-5573.gf0ed7046.el9.x86_ 16 MB/s | 18 MB 00:01 2023-08-22T23:11:08.644 INFO:teuthology.orchestra.run.smithi029.stdout:(5/31): ceph-mgr-modules-core-18.0.0-5573.gf0ed 630 kB/s | 242 kB 00:00 2023-08-22T23:11:08.769 INFO:teuthology.orchestra.run.smithi029.stdout:(6/31): python3-markupsafe-1.1.1-12.el9.x86_64. 317 kB/s | 32 kB 00:00 2023-08-22T23:11:08.796 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:57 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:08.820 INFO:teuthology.orchestra.run.smithi029.stdout:(7/31): python3-mako-1.1.4-6.el9.noarch.rpm 615 kB/s | 144 kB 00:00 2023-08-22T23:11:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:(8/31): python3-pytz-2021.1-5.el9.noarch.rpm 513 kB/s | 47 kB 00:00 2023-08-22T23:11:08.882 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:08.904 INFO:teuthology.orchestra.run.smithi029.stdout:(9/31): python3-toml-0.10.2-6.el9.noarch.rpm 459 kB/s | 39 kB 00:00 2023-08-22T23:11:08.935 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:08.936 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:08.936 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:08.957 INFO:teuthology.orchestra.run.smithi029.stdout:(10/31): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 241 kB 00:00 2023-08-22T23:11:09.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2023-08-22T23:11:09.080 INFO:teuthology.orchestra.run.smithi029.stdout:(11/31): ceph-mon-18.0.0-5573.gf0ed7046.el9.x86 2.5 MB/s | 4.4 MB 00:01 2023-08-22T23:11:09.141 INFO:teuthology.orchestra.run.smithi029.stdout:(12/31): python3-cryptography-36.0.1-4.el9.x86_ 4.8 MB/s | 1.1 MB 00:00 2023-08-22T23:11:09.191 INFO:teuthology.orchestra.run.smithi029.stdout:(13/31): python3-ply-3.11-14.el9.noarch.rpm 437 kB/s | 103 kB 00:00 2023-08-22T23:11:09.234 INFO:teuthology.orchestra.run.smithi029.stdout:(14/31): python3-pycparser-2.20-6.el9.noarch.rp 822 kB/s | 124 kB 00:00 2023-08-22T23:11:09.446 INFO:teuthology.orchestra.run.smithi029.stdout:(15/31): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 173 kB/s | 43 kB 00:00 2023-08-22T23:11:09.510 INFO:teuthology.orchestra.run.smithi029.stdout:(16/31): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2023-08-22T23:11:09.544 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:58 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:09.560 INFO:teuthology.orchestra.run.smithi029.stdout:(17/31): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2023-08-22T23:11:09.611 INFO:teuthology.orchestra.run.smithi029.stdout:(18/31): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2023-08-22T23:11:09.661 INFO:teuthology.orchestra.run.smithi029.stdout:(19/31): python3-cherrypy-18.6.1-2.el9.noarch.r 839 kB/s | 358 kB 00:00 2023-08-22T23:11:09.683 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:09.683 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 225 k 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:09.684 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:09.685 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 225 k 2023-08-22T23:11:09.685 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 216 k 2023-08-22T23:11:09.685 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:09.704 INFO:teuthology.orchestra.run.smithi029.stdout:(20/31): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2023-08-22T23:11:09.754 INFO:teuthology.orchestra.run.smithi029.stdout:(21/31): python3-jaraco-functools-3.5.0-2.el9.n 136 kB/s | 19 kB 00:00 2023-08-22T23:11:09.796 INFO:teuthology.orchestra.run.smithi029.stdout:(22/31): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2023-08-22T23:11:09.842 INFO:teuthology.orchestra.run.smithi029.stdout:(23/31): python3-logutils-0.3.5-21.el9.noarch.r 345 kB/s | 46 kB 00:00 2023-08-22T23:11:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:(24/31): python3-more-itertools-8.12.0-2.el9.no 587 kB/s | 79 kB 00:00 2023-08-22T23:11:09.929 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-18.0.0-5573.gf0ed7046.el9.noarch.rpm 920 kB/s | 225 kB 00:00 2023-08-22T23:11:09.929 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:09.930 INFO:teuthology.orchestra.run.smithi052.stdout:Total 914 kB/s | 225 kB 00:00 2023-08-22T23:11:09.930 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:09.931 INFO:teuthology.orchestra.run.smithi029.stdout:(25/31): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2023-08-22T23:11:09.937 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:09.937 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:09.938 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:09.938 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:09.983 INFO:teuthology.orchestra.run.smithi029.stdout:(26/31): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-08-22T23:11:10.012 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:10.023 INFO:teuthology.orchestra.run.smithi029.stdout:(27/31): python3-pyOpenSSL-21.0.0-1.el9.noarch. 668 kB/s | 90 kB 00:00 2023-08-22T23:11:10.067 INFO:teuthology.orchestra.run.smithi029.stdout:(28/31): python3-tempora-5.0.0-2.el9.noarch.rpm 264 kB/s | 36 kB 00:00 2023-08-22T23:11:10.126 INFO:teuthology.orchestra.run.smithi029.stdout:(29/31): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-08-22T23:11:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:(30/31): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.7 MB/s | 427 kB 00:00 2023-08-22T23:11:10.219 INFO:teuthology.orchestra.run.smithi029.stdout:(31/31): python3-zc-lockfile-2.0-10.el9.noarch. 132 kB/s | 20 kB 00:00 2023-08-22T23:11:10.226 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:10.226 INFO:teuthology.orchestra.run.smithi029.stdout:Total 9.3 MB/s | 29 MB 00:03 2023-08-22T23:11:10.227 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:10.268 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:10.268 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:10.407 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:10.525 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:10.645 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:10.646 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:10.950 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:11.264 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:11.264 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:11.264 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:11.264 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:11.264 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:11.265 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:11.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2023-08-22T23:11:11.807 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:11.924 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/31 2023-08-22T23:11:11.972 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/31 2023-08-22T23:11:12.023 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/31 2023-08-22T23:11:12.025 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:00 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:12.144 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/31 2023-08-22T23:11:12.165 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:12.165 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:12.165 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:11:12.167 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:12.167 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:12.167 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 141 k 2023-08-22T23:11:12.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:12.167 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:12.168 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:12.168 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:12.168 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:12.168 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 141 k 2023-08-22T23:11:12.168 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 395 k 2023-08-22T23:11:12.168 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:12.255 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/31 2023-08-22T23:11:12.331 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/31 2023-08-22T23:11:12.442 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/31 2023-08-22T23:11:12.485 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-18.0.0-5573.gf0ed70 441 kB/s | 141 kB 00:00 2023-08-22T23:11:12.485 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:12.486 INFO:teuthology.orchestra.run.smithi052.stdout:Total 439 kB/s | 141 kB 00:00 2023-08-22T23:11:12.486 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:12.495 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:12.495 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:12.518 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 8/31 2023-08-22T23:11:12.553 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:12.554 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:12.606 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 9/31 2023-08-22T23:11:12.701 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:12.811 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046. 1/1 2023-08-22T23:11:12.837 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046. 1/1 2023-08-22T23:11:12.837 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:12.837 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-08-22T23:11:12.837 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:12.846 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 10/31 2023-08-22T23:11:12.919 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 11/31 2023-08-22T23:11:12.978 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 12/31 2023-08-22T23:11:13.035 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 13/31 2023-08-22T23:11:13.089 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/31 2023-08-22T23:11:13.186 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/31 2023-08-22T23:11:13.253 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-08-22T23:11:13.303 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 17/31 2023-08-22T23:11:13.370 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 18/31 2023-08-22T23:11:13.432 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/31 2023-08-22T23:11:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 20/31 2023-08-22T23:11:13.626 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 21/31 2023-08-22T23:11:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 22/31 2023-08-22T23:11:13.783 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/31 2023-08-22T23:11:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046. 1/1 2023-08-22T23:11:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:13.910 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:13.911 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:13.957 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 24/31 2023-08-22T23:11:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-08-22T23:11:14.147 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2023-08-22T23:11:14.307 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5573.gf0ed7046.el9. 26/31 2023-08-22T23:11:14.364 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/31 2023-08-22T23:11:14.387 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 27/31 2023-08-22T23:11:14.387 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:14.387 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-08-22T23:11:14.388 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-22T23:11:14.388 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-22T23:11:14.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:14.686 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:03 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:14.772 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:14.823 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:14.824 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:14.824 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:14.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2023-08-22T23:11:15.400 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:04 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:15.537 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:15.538 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:15.538 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:15.538 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 1.7 M 2023-08-22T23:11:15.538 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 23 k 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 15 k 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2023-08-22T23:11:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-08-22T23:11:15.540 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:15.540 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:15.540 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:15.541 INFO:teuthology.orchestra.run.smithi052.stdout:Install 7 Packages 2023-08-22T23:11:15.541 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:15.542 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2023-08-22T23:11:15.542 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 8.5 M 2023-08-22T23:11:15.542 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:15.789 INFO:teuthology.orchestra.run.smithi052.stdout:(1/7): ceph-prometheus-alerts-18.0.0-5573.gf0ed 75 kB/s | 15 kB 00:00 2023-08-22T23:11:15.841 INFO:teuthology.orchestra.run.smithi052.stdout:(2/7): ceph-grafana-dashboards-18.0.0-5573.gf0e 94 kB/s | 23 kB 00:00 2023-08-22T23:11:16.014 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/31 2023-08-22T23:11:16.041 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 28/31 2023-08-22T23:11:16.041 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:16.042 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-08-22T23:11:16.042 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-22T23:11:16.042 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-22T23:11:16.042 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:16.098 INFO:teuthology.orchestra.run.smithi052.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 29 kB/s | 9.0 kB 00:00 2023-08-22T23:11:16.248 INFO:teuthology.orchestra.run.smithi052.stdout:(4/7): ceph-mgr-dashboard-18.0.0-5573.gf0ed7046 2.7 MB/s | 1.7 MB 00:00 2023-08-22T23:11:16.290 INFO:teuthology.orchestra.run.smithi052.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 160 kB/s | 31 kB 00:00 2023-08-22T23:11:16.316 INFO:teuthology.orchestra.run.smithi052.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 86 kB/s | 41 kB 00:00 2023-08-22T23:11:16.531 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/31 2023-08-22T23:11:16.556 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 29/31 2023-08-22T23:11:16.557 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:16.557 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-08-22T23:11:16.557 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-22T23:11:16.557 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-22T23:11:16.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:16.792 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/31 2023-08-22T23:11:16.799 INFO:teuthology.orchestra.run.smithi052.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 342 kB/s | 188 kB 00:00 2023-08-22T23:11:16.803 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:16.804 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-08-22T23:11:16.804 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:16.813 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:16.814 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:16.817 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 30/31 2023-08-22T23:11:16.817 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:16.817 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-08-22T23:11:16.817 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-22T23:11:16.818 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-22T23:11:16.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:16.875 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:16.876 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:16.900 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 31/31 2023-08-22T23:11:17.183 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:17.235 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-08-22T23:11:17.268 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-08-22T23:11:17.325 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-08-22T23:11:17.354 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-08-22T23:11:17.383 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5573.gf0ed7046.el9.n 5/7 2023-08-22T23:11:17.638 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5573.gf0ed7046.el9. 6/7 2023-08-22T23:11:17.670 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarc 7/7 2023-08-22T23:11:18.213 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarc 7/7 2023-08-22T23:11:18.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5573.gf0ed7046.el9. 1/7 2023-08-22T23:11:18.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarc 2/7 2023-08-22T23:11:18.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5573.gf0ed7046.el9.n 3/7 2023-08-22T23:11:18.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-08-22T23:11:18.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-08-22T23:11:18.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-08-22T23:11:18.543 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-08-22T23:11:18.543 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:18.544 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:18.544 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:18.544 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:18.544 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:18.545 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-08-22T23:11:18.545 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-08-22T23:11:18.545 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-08-22T23:11:18.545 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2023-08-22T23:11:18.545 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:18.545 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:18.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2023-08-22T23:11:19.286 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:08 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:19.426 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:19.427 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2023-08-22T23:11:19.427 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:19.427 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2023-08-22T23:11:19.428 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:19.428 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 7.4 M 2023-08-22T23:11:19.428 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:11:19.428 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-08-22T23:11:19.428 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-08-22T23:11:19.428 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-08-22T23:11:19.429 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-2.1.el9 CentOS-BaseOS 807 k 2023-08-22T23:11:19.429 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-2.1.el9 CentOS-BaseOS 197 k 2023-08-22T23:11:19.429 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-08-22T23:11:19.429 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-08-22T23:11:19.430 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.16-1.el9 appstream 247 k 2023-08-22T23:11:19.430 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-08-22T23:11:19.430 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-08-22T23:11:19.430 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-08-22T23:11:19.430 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:19.431 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:19.431 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2023-08-22T23:11:19.431 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2023-08-22T23:11:19.431 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:19.432 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 37 M 2023-08-22T23:11:19.432 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2023-08-22T23:11:19.433 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:19.606 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2023-08-22T23:11:19.640 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 531 kB/s | 17 kB 00:00 2023-08-22T23:11:19.682 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 834 kB/s | 34 kB 00:00 2023-08-22T23:11:19.757 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-08-22T23:11:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 31/31 2023-08-22T23:11:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/31 2023-08-22T23:11:19.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 3/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 4/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 5/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5573.gf0ed7046.el9. 6/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/31 2023-08-22T23:11:19.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/31 2023-08-22T23:11:19.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/31 2023-08-22T23:11:19.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/31 2023-08-22T23:11:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/31 2023-08-22T23:11:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/31 2023-08-22T23:11:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 15/31 2023-08-22T23:11:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-08-22T23:11:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 17/31 2023-08-22T23:11:19.784 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 18/31 2023-08-22T23:11:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 19/31 2023-08-22T23:11:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 20/31 2023-08-22T23:11:19.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 21/31 2023-08-22T23:11:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 22/31 2023-08-22T23:11:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 23/31 2023-08-22T23:11:19.786 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 24/31 2023-08-22T23:11:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-08-22T23:11:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 26/31 2023-08-22T23:11:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/31 2023-08-22T23:11:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 28/31 2023-08-22T23:11:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 29/31 2023-08-22T23:11:19.787 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 30/31 2023-08-22T23:11:19.941 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-08-22T23:11:20.024 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2023-08-22T23:11:20.050 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-08-22T23:11:20.266 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 3.7 MB/s | 807 kB 00:00 2023-08-22T23:11:20.392 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): ceph-mgr-diskprediction-local-18.0.0-55 8.0 MB/s | 7.4 MB 00:00 2023-08-22T23:11:20.551 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2023-08-22T23:11:20.576 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 644 kB/s | 197 kB 00:00 2023-08-22T23:11:20.709 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-devel-3.9.16-1.el9.x86_64.rpm 786 kB/s | 247 kB 00:00 2023-08-22T23:11:20.712 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:20.712 INFO:teuthology.orchestra.run.smithi052.stdout:Total 29 MB/s | 37 MB 00:01 2023-08-22T23:11:20.732 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:20.761 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:20.761 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:20.856 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 31/31 2023-08-22T23:11:20.856 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:20.857 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-08-22T23:11:20.858 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-08-22T23:11:20.859 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-08-22T23:11:20.860 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-08-22T23:11:20.860 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2023-08-22T23:11:20.862 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2023-08-22T23:11:20.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-08-22T23:11:20.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2023-08-22T23:11:20.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2023-08-22T23:11:20.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-08-22T23:11:20.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2023-08-22T23:11:20.863 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2023-08-22T23:11:20.865 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-08-22T23:11:20.865 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-08-22T23:11:20.865 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:20.865 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:21.187 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:21.187 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:21.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2023-08-22T23:11:21.830 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:05 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:21.917 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:21.972 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:21.973 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:11:21.974 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-08-22T23:11:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-08-22T23:11:22.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2023-08-22T23:11:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-08-22T23:11:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.16-1.el9.x86_64 4/12 2023-08-22T23:11:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-08-22T23:11:22.383 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-08-22T23:11:22.469 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-08-22T23:11:22.582 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:06 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:22.734 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:22.735 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:22.735 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:22.735 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:22.735 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:22.736 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 225 k 2023-08-22T23:11:22.736 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:22.736 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:22.736 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:22.736 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:11:22.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:22.737 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 225 k 2023-08-22T23:11:22.737 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 216 k 2023-08-22T23:11:22.737 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:23.114 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-18.0.0-5573.gf0ed7046.el9.noarch.rpm 598 kB/s | 225 kB 00:00 2023-08-22T23:11:23.114 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:23.114 INFO:teuthology.orchestra.run.smithi029.stdout:Total 596 kB/s | 225 kB 00:00 2023-08-22T23:11:23.115 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:23.119 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:23.120 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:23.121 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:23.121 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:23.159 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-08-22T23:11:23.196 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-08-22T23:11:23.705 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:24.243 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:24.593 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:24.594 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-08-22T23:11:24.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2023-08-22T23:11:25.428 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:09 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:25.566 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:25.567 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:25.567 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:11:25.568 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:25.568 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:25.568 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 141 k 2023-08-22T23:11:25.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:25.569 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:25.569 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:25.570 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:11:25.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:25.570 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 141 k 2023-08-22T23:11:25.570 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 395 k 2023-08-22T23:11:25.571 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:25.678 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-08-22T23:11:25.724 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7 12/12 2023-08-22T23:11:25.889 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-18.0.0-5573.gf0ed70 440 kB/s | 141 kB 00:00 2023-08-22T23:11:25.889 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:25.890 INFO:teuthology.orchestra.run.smithi029.stdout:Total 438 kB/s | 141 kB 00:00 2023-08-22T23:11:25.890 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:25.898 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:25.899 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:25.957 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:25.957 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:26.104 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:26.248 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046. 1/1 2023-08-22T23:11:26.274 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046. 1/1 2023-08-22T23:11:26.274 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:26.274 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-08-22T23:11:26.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:27.542 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046. 1/1 2023-08-22T23:11:27.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:27.542 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:27.542 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:27.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:27.543 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:27.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2023-08-22T23:11:28.346 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:12 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:28.424 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7 12/12 2023-08-22T23:11:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7 1/12 2023-08-22T23:11:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-08-22T23:11:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-08-22T23:11:28.435 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-08-22T23:11:28.435 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-08-22T23:11:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-08-22T23:11:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-08-22T23:11:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-08-22T23:11:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-08-22T23:11:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 10/12 2023-08-22T23:11:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 11/12 2023-08-22T23:11:28.473 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:28.474 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:11:28.474 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:28.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2023-08-22T23:11:28.942 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.16-1.el9.x86_64 12/12 2023-08-22T23:11:28.943 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:28.943 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:28.943 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:28.944 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-08-22T23:11:28.944 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-08-22T23:11:28.944 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-08-22T23:11:28.944 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-08-22T23:11:28.945 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-08-22T23:11:28.945 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2023-08-22T23:11:28.945 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-08-22T23:11:28.945 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.16-1.el9.x86_64 2023-08-22T23:11:28.945 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-08-22T23:11:28.946 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-08-22T23:11:28.946 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-08-22T23:11:28.946 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:28.946 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:29.057 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:13 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:29.195 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:29.195 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:29.196 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:29.196 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 1.7 M 2023-08-22T23:11:29.196 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:11:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 23 k 2023-08-22T23:11:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 15 k 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:29.197 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:29.198 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:29.198 INFO:teuthology.orchestra.run.smithi029.stdout:Install 7 Packages 2023-08-22T23:11:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:29.198 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2023-08-22T23:11:29.198 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 8.5 M 2023-08-22T23:11:29.198 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:29.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2023-08-22T23:11:29.450 INFO:teuthology.orchestra.run.smithi029.stdout:(1/7): ceph-grafana-dashboards-18.0.0-5573.gf0e 114 kB/s | 23 kB 00:00 2023-08-22T23:11:29.550 INFO:teuthology.orchestra.run.smithi029.stdout:(2/7): ceph-prometheus-alerts-18.0.0-5573.gf0ed 48 kB/s | 15 kB 00:00 2023-08-22T23:11:29.668 INFO:teuthology.orchestra.run.smithi029.stdout:(3/7): ceph-mgr-dashboard-18.0.0-5573.gf0ed7046 4.2 MB/s | 1.7 MB 00:00 2023-08-22T23:11:29.693 INFO:teuthology.orchestra.run.smithi029.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 37 kB/s | 9.0 kB 00:00 2023-08-22T23:11:29.852 INFO:teuthology.orchestra.run.smithi029.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 167 kB/s | 31 kB 00:00 2023-08-22T23:11:29.902 INFO:teuthology.orchestra.run.smithi029.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 116 kB/s | 41 kB 00:00 2023-08-22T23:11:29.905 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:18 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:30.045 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:30.048 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:11:30.049 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:30.049 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:11:30.049 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:30.049 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 47 k 2023-08-22T23:11:30.049 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-08-22T23:11:30.050 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:30.051 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:11:30.052 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2023-08-22T23:11:30.052 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:30.052 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2023-08-22T23:11:30.057 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2023-08-22T23:11:30.057 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:30.170 INFO:teuthology.orchestra.run.smithi029.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 396 kB/s | 188 kB 00:00 2023-08-22T23:11:30.176 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:30.176 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2023-08-22T23:11:30.177 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:30.182 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:30.182 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:30.246 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:30.246 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:30.350 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 24 kB 00:00 2023-08-22T23:11:30.375 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2023-08-22T23:11:30.417 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-08-22T23:11:30.442 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-08-22T23:11:30.467 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-08-22T23:11:30.501 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 734 kB/s | 43 kB 00:00 2023-08-22T23:11:30.536 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): ceph-mgr-rook-18.0.0-5573.gf0ed7046.el9 146 kB/s | 47 kB 00:00 2023-08-22T23:11:30.571 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:30.663 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-08-22T23:11:30.695 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 142 kB/s | 32 kB 00:00 2023-08-22T23:11:30.723 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-08-22T23:11:30.752 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2023-08-22T23:11:30.797 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-08-22T23:11:30.845 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2023-08-22T23:11:30.865 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-08-22T23:11:30.869 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-google-auth-2.22.0-1.el9.noarc 688 kB/s | 197 kB 00:00 2023-08-22T23:11:30.911 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2023-08-22T23:11:30.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5573.gf0ed7046.el9.n 5/7 2023-08-22T23:11:30.928 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2023-08-22T23:11:30.934 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:30.935 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2023-08-22T23:11:30.935 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:30.947 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:30.948 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:31.045 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:31.045 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:31.181 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5573.gf0ed7046.el9. 6/7 2023-08-22T23:11:31.237 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarc 7/7 2023-08-22T23:11:31.566 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:31.759 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-08-22T23:11:31.815 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-08-22T23:11:31.871 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-08-22T23:11:31.906 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-08-22T23:11:31.959 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-08-22T23:11:31.976 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarc 7/7 2023-08-22T23:11:31.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5573.gf0ed7046.el9. 1/7 2023-08-22T23:11:31.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarc 2/7 2023-08-22T23:11:31.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5573.gf0ed7046.el9.n 3/7 2023-08-22T23:11:31.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-08-22T23:11:31.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-08-22T23:11:31.977 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-08-22T23:11:32.047 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-08-22T23:11:32.164 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 7/13 2023-08-22T23:11:32.226 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-08-22T23:11:32.325 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-08-22T23:11:32.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:32.326 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:32.326 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:32.326 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:32.326 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:32.326 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-08-22T23:11:32.326 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-08-22T23:11:32.327 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-08-22T23:11:32.327 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2023-08-22T23:11:32.327 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:32.327 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:32.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2023-08-22T23:11:33.003 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-08-22T23:11:33.047 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-08-22T23:11:33.080 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-08-22T23:11:33.126 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-08-22T23:11:33.149 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:17 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:33.149 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 13/13 2023-08-22T23:11:33.286 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:33.287 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-08-22T23:11:33.287 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:33.287 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 7.4 M 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-08-22T23:11:33.288 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-2.1.el9 CentOS-BaseOS 807 k 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-2.1.el9 CentOS-BaseOS 197 k 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.16-1.el9 appstream 247 k 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-08-22T23:11:33.289 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-08-22T23:11:33.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:33.290 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:33.290 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-08-22T23:11:33.290 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2023-08-22T23:11:33.290 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:33.291 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2023-08-22T23:11:33.291 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2023-08-22T23:11:33.291 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2023-08-22T23:11:33.603 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2023-08-22T23:11:33.637 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-08-22T23:11:33.746 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2023-08-22T23:11:33.955 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-08-22T23:11:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 13/13 2023-08-22T23:11:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 1/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-08-22T23:11:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-08-22T23:11:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-08-22T23:11:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 10/13 2023-08-22T23:11:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-08-22T23:11:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-08-22T23:11:34.099 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2023-08-22T23:11:34.156 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.1 MB/s | 217 kB 00:00 2023-08-22T23:11:34.349 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-08-22T23:11:34.349 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:34.350 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:34.351 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:34.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-08-22T23:11:34.352 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-08-22T23:11:34.353 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-08-22T23:11:34.353 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-08-22T23:11:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-08-22T23:11:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-08-22T23:11:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-08-22T23:11:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-08-22T23:11:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-08-22T23:11:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-08-22T23:11:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2023-08-22T23:11:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-08-22T23:11:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:34.357 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:34.607 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2023-08-22T23:11:34.624 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2023-08-22T23:11:34.666 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 1.6 MB/s | 807 kB 00:00 2023-08-22T23:11:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 1.8 MB/s | 197 kB 00:00 2023-08-22T23:11:34.892 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): ceph-mgr-diskprediction-local-18.0.0-5 5.1 MB/s | 7.4 MB 00:01 2023-08-22T23:11:35.086 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): python3-devel-3.9.16-1.el9.x86_64.rpm 590 kB/s | 247 kB 00:00 2023-08-22T23:11:35.088 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:Total 21 MB/s | 37 MB 00:01 2023-08-22T23:11:35.108 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:35.136 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:35.136 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:35.174 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:24 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:35.312 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:35.313 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:11:35.313 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:35.313 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:11:35.313 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 132 k 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-08-22T23:11:35.314 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-08-22T23:11:35.315 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:35.315 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:35.315 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:11:35.315 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2023-08-22T23:11:35.315 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:35.317 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.8 M 2023-08-22T23:11:35.317 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2023-08-22T23:11:35.318 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:35.564 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:35.565 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:35.572 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-08-22T23:11:35.739 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-08-22T23:11:35.839 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): ceph-mgr-cephadm-18.0.0-5573.gf0ed7046.e 297 kB/s | 132 kB 00:00 2023-08-22T23:11:35.898 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2023-08-22T23:11:35.923 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 314 kB/s | 58 kB 00:00 2023-08-22T23:11:36.032 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 265 kB/s | 51 kB 00:00 2023-08-22T23:11:36.034 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:36.035 INFO:teuthology.orchestra.run.smithi052.stdout:Total 9.4 MB/s | 6.8 MB 00:00 2023-08-22T23:11:36.035 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:36.045 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:36.045 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:36.138 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:36.138 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:36.279 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:36.386 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-08-22T23:11:36.441 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-08-22T23:11:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:36.567 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-08-22T23:11:36.628 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-08-22T23:11:36.629 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.16-1.el9.x86_64 4/12 2023-08-22T23:11:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-08-22T23:11:36.831 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-08-22T23:11:36.887 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-08-22T23:11:37.031 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-08-22T23:11:37.287 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-08-22T23:11:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-08-22T23:11:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-08-22T23:11:37.496 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 6/6 2023-08-22T23:11:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-08-22T23:11:37.913 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-08-22T23:11:38.278 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 6/6 2023-08-22T23:11:38.279 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/6 2023-08-22T23:11:38.279 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-08-22T23:11:38.279 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-08-22T23:11:38.279 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-08-22T23:11:38.279 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-08-22T23:11:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-08-22T23:11:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:38.593 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-08-22T23:11:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2023-08-22T23:11:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-08-22T23:11:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-08-22T23:11:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-08-22T23:11:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:38.594 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:38.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2023-08-22T23:11:39.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-08-22T23:11:39.419 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:28 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:39.560 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:39.560 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:39.560 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:39.560 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 849 k 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:39.561 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2023-08-22T23:11:39.562 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:39.562 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 928 k 2023-08-22T23:11:39.562 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.7 M 2023-08-22T23:11:39.563 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:39.692 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 601 kB/s | 79 kB 00:00 2023-08-22T23:11:39.885 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-18.0.0-5573.gf0ed7046.el9.x86_ 2.6 MB/s | 849 kB 00:00 2023-08-22T23:11:39.886 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:39.886 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.8 MB/s | 928 kB 00:00 2023-08-22T23:11:39.886 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:39.912 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:39.913 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:39.990 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:39.991 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:40.177 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:40.352 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-08-22T23:11:40.376 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-08-22T23:11:40.429 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7 12/12 2023-08-22T23:11:40.442 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/2 2023-08-22T23:11:41.158 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/2 2023-08-22T23:11:41.159 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/2 2023-08-22T23:11:41.461 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-08-22T23:11:41.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:41.461 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:41.462 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-08-22T23:11:41.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:41.462 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:41.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2023-08-22T23:11:42.325 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:31 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:42.464 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:42.465 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:42.465 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2023-08-22T23:11:42.465 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 261 k 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:42.466 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:42.467 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 261 k 2023-08-22T23:11:42.467 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.2 M 2023-08-22T23:11:42.467 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:42.688 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-volume-18.0.0-5573.gf0ed7046.el9.noarch.rp 1.2 MB/s | 261 kB 00:00 2023-08-22T23:11:42.688 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:42.689 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.1 MB/s | 261 kB 00:00 2023-08-22T23:11:42.689 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:42.698 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:42.699 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:42.728 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:42.728 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:42.944 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:43.015 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:43.033 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:43.034 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:43.034 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-08-22T23:11:43.034 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:44.077 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:11:44.077 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:44.078 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:44.078 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:44.078 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:44.078 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:44.400 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2023-08-22T23:11:44.809 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7 12/12 2023-08-22T23:11:44.810 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7 1/12 2023-08-22T23:11:44.810 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-08-22T23:11:44.810 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-08-22T23:11:44.810 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-08-22T23:11:44.810 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-08-22T23:11:44.811 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-08-22T23:11:44.811 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-08-22T23:11:44.811 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-08-22T23:11:44.811 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-08-22T23:11:44.811 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 10/12 2023-08-22T23:11:44.811 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 11/12 2023-08-22T23:11:44.940 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:33 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:45.078 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:45.078 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:45.079 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:11:45.079 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:45.079 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:45.079 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 126 k 2023-08-22T23:11:45.079 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:45.079 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:45.080 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:45.080 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:45.080 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:45.080 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 126 k 2023-08-22T23:11:45.080 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2023-08-22T23:11:45.080 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:45.278 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-18.0.0-5573.gf0ed7046.el9.x86_64 636 kB/s | 126 kB 00:00 2023-08-22T23:11:45.279 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:45.279 INFO:teuthology.orchestra.run.smithi052.stdout:Total 631 kB/s | 126 kB 00:00 2023-08-22T23:11:45.279 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:45.285 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:45.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.16-1.el9.x86_64 12/12 2023-08-22T23:11:45.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-08-22T23:11:45.318 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.16-1.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-08-22T23:11:45.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:45.320 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:45.346 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:45.347 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:45.505 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:45.610 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:45.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2023-08-22T23:11:46.018 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:46.306 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:46.306 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:46.307 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:46.309 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:46.309 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:46.309 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:46.343 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:30 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:46.487 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:46.487 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:11:46.487 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:46.488 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:11:46.488 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:46.488 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 47 k 2023-08-22T23:11:46.488 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:11:46.488 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-08-22T23:11:46.488 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-08-22T23:11:46.489 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2023-08-22T23:11:46.490 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:46.491 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2023-08-22T23:11:46.491 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2023-08-22T23:11:46.491 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:46.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2023-08-22T23:11:46.778 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 18 kB 00:00 2023-08-22T23:11:46.803 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2023-08-22T23:11:46.879 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2023-08-22T23:11:46.905 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): ceph-mgr-rook-18.0.0-5573.gf0ed7046.el9 187 kB/s | 47 kB 00:00 2023-08-22T23:11:46.955 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 875 kB/s | 132 kB 00:00 2023-08-22T23:11:47.006 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 211 kB 00:00 2023-08-22T23:11:47.031 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 565 kB/s | 43 kB 00:00 2023-08-22T23:11:47.113 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:35 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:47.197 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:47.256 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:47.257 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:47.259 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:47.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2023-08-22T23:11:47.396 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 39 kB/s | 15 kB 00:00 2023-08-22T23:11:47.417 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2023-08-22T23:11:47.643 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-2.22.0-1.el9.noarc 322 kB/s | 197 kB 00:00 2023-08-22T23:11:47.760 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 172 kB/s | 59 kB 00:00 2023-08-22T23:11:47.862 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:36 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:47.911 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2023-08-22T23:11:47.953 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 290 kB/s | 90 kB 00:00 2023-08-22T23:11:47.958 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-08-22T23:11:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:47.975 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:47.975 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:47.999 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:48.000 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:48.001 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:11:48.001 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:48.001 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:48.001 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 31 k 2023-08-22T23:11:48.002 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:48.002 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:48.002 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:48.002 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:48.002 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:48.003 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 31 k 2023-08-22T23:11:48.003 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 140 k 2023-08-22T23:11:48.003 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:48.078 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:48.079 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:48.266 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-18.0.0-5573.gf0ed7046.el9.x86_6 118 kB/s | 31 kB 00:00 2023-08-22T23:11:48.266 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:48.266 INFO:teuthology.orchestra.run.smithi052.stdout:Total 118 kB/s | 31 kB 00:00 2023-08-22T23:11:48.267 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:48.270 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:48.270 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:48.289 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:48.289 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:48.507 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:48.603 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:48.810 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-08-22T23:11:48.874 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-08-22T23:11:48.951 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:48.966 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-08-22T23:11:49.036 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-08-22T23:11:49.104 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-08-22T23:11:49.210 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:49.210 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:49.210 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:49.210 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:49.211 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:49.211 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:49.226 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-08-22T23:11:49.363 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 7/13 2023-08-22T23:11:49.419 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2023-08-22T23:11:49.442 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-08-22T23:11:49.956 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:38 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:50.041 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:50.093 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:50.094 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:50.094 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:50.168 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2023-08-22T23:11:50.246 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-08-22T23:11:50.359 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-08-22T23:11:50.412 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-08-22T23:11:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-08-22T23:11:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 13/13 2023-08-22T23:11:50.701 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:39 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:50.790 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:50.843 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:50.843 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:50.844 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:50.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2023-08-22T23:11:51.453 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:40 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:51.539 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:51.590 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:51.591 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:51.591 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:51.653 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 13/13 2023-08-22T23:11:51.654 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 1/13 2023-08-22T23:11:51.654 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-08-22T23:11:51.654 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-08-22T23:11:51.654 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-08-22T23:11:51.654 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-08-22T23:11:51.654 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-08-22T23:11:51.655 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-08-22T23:11:51.655 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-08-22T23:11:51.655 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-08-22T23:11:51.655 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 10/13 2023-08-22T23:11:51.655 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-08-22T23:11:51.655 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-08-22T23:11:51.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2023-08-22T23:11:52.058 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-08-22T23:11:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:52.059 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:52.059 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-08-22T23:11:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-08-22T23:11:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-08-22T23:11:52.060 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-08-22T23:11:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-08-22T23:11:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-08-22T23:11:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-08-22T23:11:52.061 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-08-22T23:11:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-08-22T23:11:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-08-22T23:11:52.062 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2023-08-22T23:11:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-08-22T23:11:52.063 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:52.064 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:52.197 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:41 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:52.283 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:52.337 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:52.338 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:52.338 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:52.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2023-08-22T23:11:52.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2023-08-22T23:11:52.947 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:36 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:52.948 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:41 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:53.035 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:53.087 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:53.088 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:53.088 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:11:53.088 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:53.088 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 132 k 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-08-22T23:11:53.089 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2023-08-22T23:11:53.090 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:53.091 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:53.091 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:53.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2023-08-22T23:11:53.612 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.2 MB/s | 228 kB 00:00 2023-08-22T23:11:53.697 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:42 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:53.782 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:11:53.796 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-08-22T23:11:53.834 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:53.835 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:11:53.836 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:53.890 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): ceph-mgr-cephadm-18.0.0-5573.gf0ed7046.e 282 kB/s | 132 kB 00:00 2023-08-22T23:11:53.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2023-08-22T23:11:53.957 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 359 kB/s | 58 kB 00:00 2023-08-22T23:11:54.016 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2023-08-22T23:11:54.041 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 338 kB/s | 51 kB 00:00 2023-08-22T23:11:54.043 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:54.044 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2023-08-22T23:11:54.044 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:54.053 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:54.053 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:54.146 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:54.146 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:54.438 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:43 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:54.451 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:54.577 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:54.577 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2023-08-22T23:11:54.577 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:54.577 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 85 k 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:54.578 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 85 k 2023-08-22T23:11:54.579 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 227 k 2023-08-22T23:11:54.579 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:54.664 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-08-22T23:11:54.747 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-08-22T23:11:54.902 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-18.0.0-5573.gf0ed7046.el9.x86_64.rpm 261 kB/s | 85 kB 00:00 2023-08-22T23:11:54.904 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:54.904 INFO:teuthology.orchestra.run.smithi052.stdout:Total 260 kB/s | 85 kB 00:00 2023-08-22T23:11:54.905 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:54.911 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:54.979 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:54.979 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:55.131 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:55.226 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:55.332 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-08-22T23:11:55.462 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-08-22T23:11:55.559 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-08-22T23:11:55.588 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:55.607 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 6/6 2023-08-22T23:11:55.891 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:55.892 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:55.892 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:55.892 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:55.892 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:55.892 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:56.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2023-08-22T23:11:56.716 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:45 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:11:56.781 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 6/6 2023-08-22T23:11:56.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 1/6 2023-08-22T23:11:56.781 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-08-22T23:11:56.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-08-22T23:11:56.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-08-22T23:11:56.782 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-08-22T23:11:56.857 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:11:56.858 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:56.858 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:11:56.858 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:56.858 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:11:56.858 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 3.0 M 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.0 M 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 12 M 2023-08-22T23:11:56.859 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:11:57.139 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-08-22T23:11:57.140 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:57.140 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:11:57.140 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:11:57.140 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-08-22T23:11:57.140 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2023-08-22T23:11:57.140 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-08-22T23:11:57.141 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-08-22T23:11:57.141 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-08-22T23:11:57.141 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:57.141 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:11:57.466 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-18.0.0-5573.gf0ed7046.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2023-08-22T23:11:57.466 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:57.466 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.0 MB/s | 3.0 MB 00:00 2023-08-22T23:11:57.467 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:11:57.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2023-08-22T23:11:57.476 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:11:57.476 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:11:57.555 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:11:57.555 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:11:57.993 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:11:58.008 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:42 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:11:58.065 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:58.091 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:58.091 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:11:58.092 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-08-22T23:11:58.092 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-22T23:11:58.092 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-22T23:11:58.092 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:58.147 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:11:58.148 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:58.148 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:11:58.148 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:58.148 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:11:58.148 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 849 k 2023-08-22T23:11:58.149 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:11:58.149 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-08-22T23:11:58.149 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:58.153 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:11:58.154 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:11:58.154 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2023-08-22T23:11:58.154 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:11:58.155 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 928 k 2023-08-22T23:11:58.155 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.7 M 2023-08-22T23:11:58.155 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:11:58.324 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 450 kB/s | 79 kB 00:00 2023-08-22T23:11:58.833 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-18.0.0-5573.gf0ed7046.el9.x86_ 1.2 MB/s | 849 kB 00:00 2023-08-22T23:11:58.834 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:11:58.834 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.3 MB/s | 928 kB 00:00 2023-08-22T23:11:58.834 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:11:58.858 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:11:58.858 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:11:58.937 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:11:58.937 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:11:59.121 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:11:59.286 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-08-22T23:11:59.359 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:11:59.359 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:59.359 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:11:59.359 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:11:59.360 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:11:59.360 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:11:59.378 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/2 2023-08-22T23:11:59.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2023-08-22T23:12:00.144 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:49 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:12:00.199 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/2 2023-08-22T23:12:00.199 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/2 2023-08-22T23:12:00.282 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:12:00.283 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 168 k 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:00.284 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:12:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:00.285 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 168 k 2023-08-22T23:12:00.285 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 486 k 2023-08-22T23:12:00.285 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:12:00.485 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-18.0.0-5573.gf0ed7046.el9.x86_64.rpm 836 kB/s | 168 kB 00:00 2023-08-22T23:12:00.486 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:00.486 INFO:teuthology.orchestra.run.smithi052.stdout:Total 830 kB/s | 168 kB 00:00 2023-08-22T23:12:00.487 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:12:00.495 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:12:00.496 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:12:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-08-22T23:12:00.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:00.556 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-08-22T23:12:00.556 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:00.556 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:00.557 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:12:00.558 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:12:00.710 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:12:00.852 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:00.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2023-08-22T23:12:01.251 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:01.456 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:45 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:01.535 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:01.536 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:01.536 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:12:01.536 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:01.537 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:01.537 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:12:01.595 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:01.595 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:01.595 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-08-22T23:12:01.596 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:01.596 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:01.596 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:18.0.0-5573.gf0ed7046.el9 ceph-noarch 261 k 2023-08-22T23:12:01.596 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:01.596 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:01.596 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:01.597 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:01.597 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:01.597 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 261 k 2023-08-22T23:12:01.597 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.2 M 2023-08-22T23:12:01.598 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:01.839 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base-debuginfo 2023-08-22T23:12:01.996 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-volume-18.0.0-5573.gf0ed7046.el9.noarch.rp 654 kB/s | 261 kB 00:00 2023-08-22T23:12:01.997 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:01.998 INFO:teuthology.orchestra.run.smithi029.stdout:Total 651 kB/s | 261 kB 00:00 2023-08-22T23:12:01.998 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:02.006 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:02.007 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:02.042 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:02.043 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:02.265 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:02.383 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:51 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:12:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:12:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:12:02.453 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:12:02.453 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-08-22T23:12:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:02.526 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:12:02.526 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:02.526 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 74 M 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 20 M 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:02.527 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:12:02.528 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:02.528 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2023-08-22T23:12:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:02.528 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 94 M 2023-08-22T23:12:02.529 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 510 M 2023-08-22T23:12:02.529 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:12:03.680 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 1/1 2023-08-22T23:12:03.680 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:03.681 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:03.681 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:18.0.0-5573.gf0ed7046.el9.noarch 2023-08-22T23:12:03.681 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:03.681 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:04.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2023-08-22T23:12:04.577 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:48 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:04.713 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:04.714 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:04.714 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 126 k 2023-08-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:04.716 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:04.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:04.716 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 126 k 2023-08-22T23:12:04.716 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 456 k 2023-08-22T23:12:04.717 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:05.154 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-18.0.0-5573.gf0ed7046.el9.x86_64 287 kB/s | 126 kB 00:00 2023-08-22T23:12:05.155 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:05.156 INFO:teuthology.orchestra.run.smithi029.stdout:Total 286 kB/s | 126 kB 00:00 2023-08-22T23:12:05.156 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:05.162 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:05.162 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:05.224 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:05.224 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:05.376 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:05.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:06.070 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:06.429 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:06.429 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:06.431 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:06.431 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:06.431 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:06.431 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:06.631 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): ceph-debuginfo-18.0.0-5573.gf0ed7046.el9 5.0 MB/s | 20 MB 00:04 2023-08-22T23:12:06.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2023-08-22T23:12:07.269 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:51 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:07.353 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:07.406 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:07.406 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:07.407 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:07.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2023-08-22T23:12:08.059 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:52 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:08.201 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:08.201 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:08.201 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 31 k 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:08.202 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:08.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:08.203 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 k 2023-08-22T23:12:08.203 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 140 k 2023-08-22T23:12:08.203 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:08.435 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-18.0.0-5573.gf0ed7046.el9.x86_6 132 kB/s | 31 kB 00:00 2023-08-22T23:12:08.436 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:08.436 INFO:teuthology.orchestra.run.smithi029.stdout:Total 131 kB/s | 31 kB 00:00 2023-08-22T23:12:08.436 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:08.440 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:08.440 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:08.460 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:08.461 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:09.490 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:09.490 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:09.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2023-08-22T23:12:09.873 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-base-debuginfo-18.0.0-5573.gf0ed704 10 MB/s | 74 MB 00:07 2023-08-22T23:12:09.875 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:09.876 INFO:teuthology.orchestra.run.smithi052.stdout:Total 13 MB/s | 94 MB 00:07 2023-08-22T23:12:09.876 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:12:09.882 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:12:09.882 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:12:10.296 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:54 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:10.384 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:10.436 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:10.439 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:10.439 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:10.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2023-08-22T23:12:10.513 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:12:10.514 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:12:11.041 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:55 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:11.128 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:11.181 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:11.182 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:11.182 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:11.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2023-08-22T23:12:11.798 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:55 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:11.886 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:11.935 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:11.936 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:11.936 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:12.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2023-08-22T23:12:12.563 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:56 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:12.650 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:12.707 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:12.707 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:12.707 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:12.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2023-08-22T23:12:13.320 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:57 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:13.406 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:13.460 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:13.460 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:13.460 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:13.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2023-08-22T23:12:13.572 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:12:14.082 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:58 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:14.168 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:12:14.219 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:14.220 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:12:14.220 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:14.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2023-08-22T23:12:14.843 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:58 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:14.981 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:14.982 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:14.982 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-08-22T23:12:14.982 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:14.982 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:14.982 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 85 k 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 85 k 2023-08-22T23:12:14.983 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 227 k 2023-08-22T23:12:14.984 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-18.0.0-5573.gf0ed7046.el9.x86_64.rpm 375 kB/s | 85 kB 00:00 2023-08-22T23:12:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:Total 373 kB/s | 85 kB 00:00 2023-08-22T23:12:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:15.218 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:15.218 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:15.287 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:15.287 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:15.444 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:15.570 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:16.010 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:16.362 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:16.362 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:16.363 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:16.363 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:16.363 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:16.363 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:16.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2023-08-22T23:12:17.219 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:01 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:17.351 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:17.353 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:17.353 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:12:17.353 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 3.0 M 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:17.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:17.355 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2023-08-22T23:12:17.355 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 12 M 2023-08-22T23:12:17.355 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:17.963 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-18.0.0-5573.gf0ed7046.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2023-08-22T23:12:17.964 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:17.964 INFO:teuthology.orchestra.run.smithi029.stdout:Total 5.0 MB/s | 3.0 MB 00:00 2023-08-22T23:12:17.964 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:17.974 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:17.974 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:18.056 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:18.057 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:18.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:18.597 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-22T23:12:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-08-22T23:12:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-22T23:12:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-22T23:12:18.598 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:19.951 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:19.951 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:19.952 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:19.952 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:19.952 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:19.952 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:20.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2023-08-22T23:12:20.819 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:04 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:20.961 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:20.961 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:20.961 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-08-22T23:12:20.961 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 168 k 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:20.962 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 168 k 2023-08-22T23:12:20.963 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 486 k 2023-08-22T23:12:20.963 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:21.348 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-18.0.0-5573.gf0ed7046.el9.x86_64.rpm 433 kB/s | 168 kB 00:00 2023-08-22T23:12:21.349 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:21.349 INFO:teuthology.orchestra.run.smithi029.stdout:Total 432 kB/s | 168 kB 00:00 2023-08-22T23:12:21.349 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:21.358 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:21.359 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:21.420 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:21.420 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:21.573 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/2 2023-08-22T23:12:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 2/2 2023-08-22T23:12:22.186 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:12:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:22.540 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:22.541 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:22.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base-debuginfo 2023-08-22T23:12:23.395 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:07 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:23.534 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:23.535 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:12:23.535 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:23.535 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:23.536 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 74 M 2023-08-22T23:12:23.536 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-08-22T23:12:23.536 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 20 M 2023-08-22T23:12:23.536 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:23.537 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:23.537 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:23.537 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2023-08-22T23:12:23.537 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 94 M 2023-08-22T23:12:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 510 M 2023-08-22T23:12:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:24.884 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): ceph-debuginfo-18.0.0-5573.gf0ed7046.el9 15 MB/s | 20 MB 00:01 2023-08-22T23:12:25.751 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 2/2 2023-08-22T23:12:25.752 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/2 2023-08-22T23:12:26.064 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/2 2023-08-22T23:12:26.065 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:26.065 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:12:26.065 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:26.065 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:26.065 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:26.065 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:12:26.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-common-debuginfo 2023-08-22T23:12:26.835 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:15 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:12:26.968 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:12:26.969 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:12:26.969 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:26.970 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:12:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 409 M 2023-08-22T23:12:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:26.970 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:12:26.970 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:12:26.971 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:12:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:12:26.971 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 409 M 2023-08-22T23:12:26.971 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.7 G 2023-08-22T23:12:26.971 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:12:28.943 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-base-debuginfo-18.0.0-5573.gf0ed704 14 MB/s | 74 MB 00:05 2023-08-22T23:12:28.944 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:28.945 INFO:teuthology.orchestra.run.smithi029.stdout:Total 17 MB/s | 94 MB 00:05 2023-08-22T23:12:28.945 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:12:28.950 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:12:28.951 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:12:29.585 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:12:29.586 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:12:32.646 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:12:40.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/2 2023-08-22T23:12:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 2/2 2023-08-22T23:12:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 2/2 2023-08-22T23:12:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/2 2023-08-22T23:12:48.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2/2 2023-08-22T23:12:48.441 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:48.441 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:12:48.441 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:48.441 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:12:48.441 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:48.441 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:12:48.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-common-debuginfo 2023-08-22T23:12:49.256 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:33 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:12:49.393 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:12:49.394 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 409 M 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:12:49.395 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:12:49.396 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:12:49.400 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 409 M 2023-08-22T23:12:49.400 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.7 G 2023-08-22T23:12:49.401 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:12:50.681 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-common-debuginfo-18.0.0-5573.gf0ed7046.el9 17 MB/s | 409 MB 00:23 2023-08-22T23:12:50.681 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:12:50.682 INFO:teuthology.orchestra.run.smithi052.stdout:Total 17 MB/s | 409 MB 00:23 2023-08-22T23:12:50.682 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:12:50.682 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:12:50.683 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:12:53.246 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:12:53.247 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:13:11.665 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-common-debuginfo-18.0.0-5573.gf0ed7046.el9 18 MB/s | 409 MB 00:22 2023-08-22T23:13:11.666 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:13:11.666 INFO:teuthology.orchestra.run.smithi029.stdout:Total 18 MB/s | 409 MB 00:22 2023-08-22T23:13:11.667 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:13:11.671 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:13:11.672 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:13:14.235 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:13:14.236 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:13:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:13:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:13:45.269 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:13:45.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:13:45.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:45.527 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:13:45.527 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:13:45.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:45.527 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:13:45.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-08-22T23:13:46.341 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:03:35 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:13:46.480 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:13:46.481 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:13:46.481 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:13:46.481 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:13:46.481 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:13:46.481 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.4 M 2023-08-22T23:13:46.481 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.4 M 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 6.3 M 2023-08-22T23:13:46.482 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:13:46.819 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-55 4.3 MB/s | 1.4 MB 00:00 2023-08-22T23:13:46.820 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:13:46.820 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.2 MB/s | 1.4 MB 00:00 2023-08-22T23:13:46.820 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:13:46.823 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:13:46.824 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:13:46.869 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:13:46.869 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:13:47.136 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:13:47.260 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-5573. 1/1 2023-08-22T23:13:47.641 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-5573. 1/1 2023-08-22T23:13:47.900 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-5573. 1/1 2023-08-22T23:13:47.901 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:47.901 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:13:47.901 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:13:47.901 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:47.902 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:13:48.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw-debuginfo 2023-08-22T23:13:48.658 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:03:37 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:13:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:13:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:13:48.798 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:13:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:13:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:13:48.798 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 169 M 2023-08-22T23:13:48.798 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:13:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:13:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:13:48.801 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:13:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 169 M 2023-08-22T23:13:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 713 M 2023-08-22T23:13:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:13:57.038 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-radosgw-debuginfo-18.0.0-5573.gf0ed7046.el 21 MB/s | 169 MB 00:08 2023-08-22T23:13:57.039 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:13:57.039 INFO:teuthology.orchestra.run.smithi052.stdout:Total 21 MB/s | 169 MB 00:08 2023-08-22T23:13:57.039 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:13:57.044 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:13:57.044 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:13:58.122 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:13:58.123 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:13:58.453 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:13:58.552 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:14:15.672 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:14:15.966 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:14:15.966 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:15.966 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:14:15.966 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:14:15.967 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:15.967 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:14:16.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-08-22T23:14:16.475 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:14:16.596 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x 1/1 2023-08-22T23:14:16.810 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:04:00 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:14:16.949 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:14:16.949 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:14:16.949 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:14:16.949 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:14:16.949 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:14:16.949 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.4 M 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.4 M 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 6.3 M 2023-08-22T23:14:16.950 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:14:17.300 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-55 4.1 MB/s | 1.4 MB 00:00 2023-08-22T23:14:17.301 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:14:17.301 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.1 MB/s | 1.4 MB 00:00 2023-08-22T23:14:17.302 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:14:17.305 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:14:17.305 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:14:17.349 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:14:17.349 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:14:17.617 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:14:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-5573. 1/1 2023-08-22T23:14:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-5573. 1/1 2023-08-22T23:14:18.554 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-5573. 1/1 2023-08-22T23:14:18.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:18.554 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:14:18.555 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:14:18.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:18.555 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:14:18.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw-debuginfo 2023-08-22T23:14:19.320 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:04:03 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:14:19.458 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:14:19.460 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:14:19.460 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:14:19.460 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 169 M 2023-08-22T23:14:19.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:19.462 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:14:19.462 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:14:19.462 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:14:19.462 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:19.463 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 169 M 2023-08-22T23:14:19.463 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 713 M 2023-08-22T23:14:19.463 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:14:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x 1/1 2023-08-22T23:14:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x 1/1 2023-08-22T23:14:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:14:21.236 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:14:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:14:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:14:21.236 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:14:21.454 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test-debuginfo 2023-08-22T23:14:21.993 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:10 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:14:22.133 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:14:22.133 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:14:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:14:22.134 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:14:22.134 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:14:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 619 M 2023-08-22T23:14:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:14:22.135 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:14:22.135 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:14:22.135 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:14:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:14:22.135 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 619 M 2023-08-22T23:14:22.136 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.8 G 2023-08-22T23:14:22.136 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:14:29.325 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-radosgw-debuginfo-18.0.0-5573.gf0ed7046.el 17 MB/s | 169 MB 00:09 2023-08-22T23:14:29.326 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:14:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:Total 17 MB/s | 169 MB 00:09 2023-08-22T23:14:29.327 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:14:29.331 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:14:29.331 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:14:30.410 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:14:30.411 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:14:48.799 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:14:48.966 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x 1/1 2023-08-22T23:14:58.469 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x 1/1 2023-08-22T23:14:58.766 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x 1/1 2023-08-22T23:14:58.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:58.767 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:14:58.767 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:14:58.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:58.767 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:14:59.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test-debuginfo 2023-08-22T23:14:59.569 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:04:43 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:14:59.707 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:14:59.707 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:14:59.708 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:14:59.708 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:14:59.708 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:14:59.708 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 619 M 2023-08-22T23:14:59.708 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:59.708 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:14:59.709 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:14:59.709 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:14:59.709 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:14:59.709 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 619 M 2023-08-22T23:14:59.709 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.8 G 2023-08-22T23:14:59.709 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:15:06.104 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-test-debuginfo-18.0.0-5573.gf0ed7046.el9.x 14 MB/s | 619 MB 00:43 2023-08-22T23:15:06.104 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:15:06.105 INFO:teuthology.orchestra.run.smithi052.stdout:Total 14 MB/s | 619 MB 00:43 2023-08-22T23:15:06.105 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:15:06.110 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:15:06.110 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:15:09.964 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:15:09.965 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:15:42.439 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-test-debuginfo-18.0.0-5573.gf0ed7046.el9.x 14 MB/s | 619 MB 00:42 2023-08-22T23:15:42.439 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:15:42.440 INFO:teuthology.orchestra.run.smithi029.stdout:Total 14 MB/s | 619 MB 00:42 2023-08-22T23:15:42.440 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:15:42.446 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:15:42.446 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:15:46.302 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:15:46.302 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:16:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:16:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:16:29.347 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:16:29.615 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:16:29.615 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:29.615 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:16:29.616 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:16:29.616 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:29.616 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:16:29.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base-debuginfo 2023-08-22T23:16:30.445 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:19 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:16:30.533 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:16:30.582 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:16:30.584 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2023-08-22T23:16:30.584 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:16:30.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-debuginfo 2023-08-22T23:16:31.196 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:20 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:16:31.334 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:16:31.335 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:16:31.335 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:31.335 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:16:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 19 M 2023-08-22T23:16:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:31.336 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:16:31.336 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:31.336 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:16:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:31.337 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 19 M 2023-08-22T23:16:31.337 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 101 M 2023-08-22T23:16:31.337 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:16:32.425 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mgr-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 18 MB/s | 19 MB 00:01 2023-08-22T23:16:32.426 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:16:32.426 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 19 MB 00:01 2023-08-22T23:16:32.426 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:16:32.430 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:16:32.431 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:16:32.586 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:16:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:16:34.910 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:16:35.023 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:16:36.073 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:16:36.347 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:16:36.347 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:36.347 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:16:36.347 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:16:36.348 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:36.348 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:16:36.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mds-debuginfo 2023-08-22T23:16:37.073 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:25 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:16:37.211 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:16:37.212 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:37.212 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:16:37.213 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:37.213 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:16:37.213 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 48 M 2023-08-22T23:16:37.213 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:37.213 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:16:37.214 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:37.214 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:16:37.214 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:37.214 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 48 M 2023-08-22T23:16:37.214 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 189 M 2023-08-22T23:16:37.215 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:16:39.703 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mds-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 19 MB/s | 48 MB 00:02 2023-08-22T23:16:39.704 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:16:39.704 INFO:teuthology.orchestra.run.smithi052.stdout:Total 19 MB/s | 48 MB 00:02 2023-08-22T23:16:39.704 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:16:39.709 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:16:39.709 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:16:40.042 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:16:40.043 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:16:45.191 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:16:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:16:46.741 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:16:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:16:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:46.999 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:16:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:16:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:47.000 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:16:47.192 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mon-debuginfo 2023-08-22T23:16:47.730 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:36 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:16:47.867 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:16:47.868 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:47.869 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:16:47.869 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:47.869 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:16:47.869 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 112 M 2023-08-22T23:16:47.869 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:16:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:16:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:16:47.870 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:16:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 112 M 2023-08-22T23:16:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 508 M 2023-08-22T23:16:47.871 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:16:55.330 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mon-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 15 MB/s | 112 MB 00:07 2023-08-22T23:16:55.331 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:16:55.331 INFO:teuthology.orchestra.run.smithi052.stdout:Total 15 MB/s | 112 MB 00:07 2023-08-22T23:16:55.331 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:16:55.336 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:16:55.336 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:16:55.681 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:16:56.063 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:16:56.063 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:16:56.334 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:17:08.432 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:17:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:12.023 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:12.278 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:12.278 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:17:12.279 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:17:12.279 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:17:12.279 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:17:12.279 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:17:12.419 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:17:12.493 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-osd-debuginfo 2023-08-22T23:17:12.725 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:17:12.725 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:12.725 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:17:12.726 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:17:12.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:12.726 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:17:13.026 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:07:01 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:17:13.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base-debuginfo 2023-08-22T23:17:13.165 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:17:13.166 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:17:13.166 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:17:13.166 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:17:13.166 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 313 M 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 313 M 2023-08-22T23:17:13.167 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.3 G 2023-08-22T23:17:13.168 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:17:13.643 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:57 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:17:13.729 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 is already installed. 2023-08-22T23:17:13.781 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:17:13.782 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-08-22T23:17:13.782 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:17:13.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-debuginfo 2023-08-22T23:17:14.401 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:58 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:17:14.538 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:17:14.538 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 19 M 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:14.539 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:17:14.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:14.540 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 19 M 2023-08-22T23:17:14.540 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 101 M 2023-08-22T23:17:14.540 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:17:23.007 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-mgr-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 2.3 MB/s | 19 MB 00:08 2023-08-22T23:17:23.007 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:17:23.008 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.3 MB/s | 19 MB 00:08 2023-08-22T23:17:23.008 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:17:23.012 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:17:23.013 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:17:23.167 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:17:23.168 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:17:25.490 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:17:25.615 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:27.535 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:27.824 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:27.825 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:27.825 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:17:27.825 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:17:27.825 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:27.825 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:17:28.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mds-debuginfo 2023-08-22T23:17:28.616 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:12 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:17:28.754 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:17:28.754 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:17:28.755 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:28.755 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:17:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 48 M 2023-08-22T23:17:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:28.756 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:17:28.756 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:28.756 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:17:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:28.757 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 48 M 2023-08-22T23:17:28.757 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 189 M 2023-08-22T23:17:28.757 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:17:29.474 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-osd-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 19 MB/s | 313 MB 00:16 2023-08-22T23:17:29.475 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:17:29.475 INFO:teuthology.orchestra.run.smithi052.stdout:Total 19 MB/s | 313 MB 00:16 2023-08-22T23:17:29.475 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:17:29.480 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:17:29.480 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:17:31.443 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:17:31.444 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:17:32.083 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-mds-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 14 MB/s | 48 MB 00:03 2023-08-22T23:17:32.084 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:17:32.084 INFO:teuthology.orchestra.run.smithi029.stdout:Total 14 MB/s | 48 MB 00:03 2023-08-22T23:17:32.084 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:17:32.088 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:17:32.089 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:17:32.421 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:17:32.422 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:17:37.586 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:17:37.725 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:40.686 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:40.989 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:17:40.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:40.989 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:17:40.990 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:17:40.990 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:40.990 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:17:41.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mon-debuginfo 2023-08-22T23:17:41.786 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:25 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:17:41.924 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:17:41.924 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 112 M 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:17:41.925 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:17:41.926 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:17:41.926 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:17:41.926 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 112 M 2023-08-22T23:17:41.926 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 508 M 2023-08-22T23:17:41.927 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:17:48.941 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-mon-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 16 MB/s | 112 MB 00:07 2023-08-22T23:17:48.941 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:17:48.942 INFO:teuthology.orchestra.run.smithi029.stdout:Total 16 MB/s | 112 MB 00:07 2023-08-22T23:17:48.942 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:17:48.946 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:17:48.947 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:17:49.673 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:17:49.673 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:18:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:18:02.265 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:18:06.016 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:06.137 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:18:09.132 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:18:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:18:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:18:09.401 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:18:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:18:09.402 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:18:09.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-osd-debuginfo 2023-08-22T23:18:10.208 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:54 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:18:10.349 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:18:10.350 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:18:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:18:10.350 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:18:10.350 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:18:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 313 M 2023-08-22T23:18:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 313 M 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.3 G 2023-08-22T23:18:10.351 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:18:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:14.278 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:14.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse-debuginfo 2023-08-22T23:18:15.043 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:03 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:15.182 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:15.183 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:15.183 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:15.183 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 5.7 M 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:15.184 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:15.185 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.7 M 2023-08-22T23:18:15.185 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 27 M 2023-08-22T23:18:15.185 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:16.724 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-fuse-debuginfo-18.0.0-5573.gf0ed7046.el9.x 3.7 MB/s | 5.7 MB 00:01 2023-08-22T23:18:16.725 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:16.725 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.7 MB/s | 5.7 MB 00:01 2023-08-22T23:18:16.725 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:16.729 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:16.730 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:16.804 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:16.805 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:17.550 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:17.682 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:18:18.280 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:18:18.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:18:18.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:18.529 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:18.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel-debuginfo 2023-08-22T23:18:19.268 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:08 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:19.409 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:19.410 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:19.410 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:19.410 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:19.411 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:19.411 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 109 k 2023-08-22T23:18:19.411 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:19.411 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:19.411 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:19.411 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:19.412 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:19.412 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 109 k 2023-08-22T23:18:19.412 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 440 k 2023-08-22T23:18:19.412 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:19.769 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-debuginfo-18.0.0-5573.gf0ed7046. 303 kB/s | 109 kB 00:00 2023-08-22T23:18:19.769 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:19.770 INFO:teuthology.orchestra.run.smithi052.stdout:Total 302 kB/s | 109 kB 00:00 2023-08-22T23:18:19.770 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:19.773 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:19.773 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:19.809 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:19.809 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:19.932 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:20.077 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:18:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:18:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:18:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:20.700 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:20.700 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:20.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2-debuginfo 2023-08-22T23:18:21.452 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:10 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:21.592 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:21.593 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:21.593 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:21.593 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 4.9 M 2023-08-22T23:18:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 4.9 M 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 23 M 2023-08-22T23:18:21.594 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:23.289 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs2-debuginfo-18.0.0-5573.gf0ed7046.el9. 2.9 MB/s | 4.9 MB 00:01 2023-08-22T23:18:23.290 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:23.290 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.9 MB/s | 4.9 MB 00:01 2023-08-22T23:18:23.290 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:23.294 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:23.295 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:23.364 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:23.365 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:24.029 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:24.164 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:18:24.657 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:18:24.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:18:24.912 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:24.912 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:24.912 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:24.912 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:24.912 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:25.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2-debuginfo 2023-08-22T23:18:25.688 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:14 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:25.826 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 44 M 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:25.827 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:25.828 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:25.828 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:25.828 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:25.828 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 44 M 2023-08-22T23:18:25.828 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 238 M 2023-08-22T23:18:25.828 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:28.225 INFO:teuthology.orchestra.run.smithi052.stdout:librados2-debuginfo-18.0.0-5573.gf0ed7046.el9.x 18 MB/s | 44 MB 00:02 2023-08-22T23:18:28.226 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:28.226 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 44 MB 00:02 2023-08-22T23:18:28.227 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:28.231 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:28.231 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:28.546 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:28.546 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:31.163 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-osd-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 15 MB/s | 313 MB 00:20 2023-08-22T23:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout:Total 15 MB/s | 313 MB 00:20 2023-08-22T23:18:31.165 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:18:33.133 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:18:33.133 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:18:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:18:35.766 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:18:36.009 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:18:36.009 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:36.010 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:36.010 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:36.010 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:36.011 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:36.232 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1-debuginfo 2023-08-22T23:18:36.766 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:25 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:36.903 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:36.905 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:36.905 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:36.905 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:36.905 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:36.905 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 51 M 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 51 M 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 299 M 2023-08-22T23:18:36.906 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:41.728 INFO:teuthology.orchestra.run.smithi052.stdout:librbd1-debuginfo-18.0.0-5573.gf0ed7046.el9.x86 11 MB/s | 51 MB 00:04 2023-08-22T23:18:41.729 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:41.730 INFO:teuthology.orchestra.run.smithi052.stdout:Total 11 MB/s | 51 MB 00:04 2023-08-22T23:18:41.730 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:41.734 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:41.734 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:42.091 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:42.092 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:48.289 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:18:50.578 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:18:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:18:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:50.862 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:50.863 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:51.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs-debuginfo 2023-08-22T23:18:51.675 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:40 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:51.813 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:51.814 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:51.814 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:51.814 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 615 k 2023-08-22T23:18:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:51.815 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:51.815 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:51.815 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:51.815 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 615 k 2023-08-22T23:18:51.816 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.1 M 2023-08-22T23:18:51.816 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:52.099 INFO:teuthology.orchestra.run.smithi052.stdout:python3-cephfs-debuginfo-18.0.0-5573.gf0ed7046. 2.1 MB/s | 615 kB 00:00 2023-08-22T23:18:52.100 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:52.100 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.1 MB/s | 615 kB 00:00 2023-08-22T23:18:52.100 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:52.103 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:52.104 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:52.144 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:52.145 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:18:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:18:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:18:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:53.098 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:53.098 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:53.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados-debuginfo 2023-08-22T23:18:53.864 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:42 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:54.005 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:54.006 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 769 k 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:54.008 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 769 k 2023-08-22T23:18:54.008 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.7 M 2023-08-22T23:18:54.008 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:54.311 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rados-debuginfo-18.0.0-5573.gf0ed7046.e 2.5 MB/s | 769 kB 00:00 2023-08-22T23:18:54.312 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:54.312 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.5 MB/s | 769 kB 00:00 2023-08-22T23:18:54.312 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:54.315 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:54.315 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:54.356 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:54.356 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:54.668 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9. 1/1 2023-08-22T23:18:55.081 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9. 1/1 2023-08-22T23:18:55.340 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9. 1/1 2023-08-22T23:18:55.340 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:55.340 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:55.341 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:55.341 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:55.341 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:55.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd-debuginfo 2023-08-22T23:18:56.105 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:44 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:56.243 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:56.244 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:56.244 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:56.244 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.1 M 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:56.245 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:56.246 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.1 M 2023-08-22T23:18:56.246 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 4.3 M 2023-08-22T23:18:56.246 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:18:56.756 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rbd-debuginfo-18.0.0-5573.gf0ed7046.el9 2.2 MB/s | 1.1 MB 00:00 2023-08-22T23:18:56.756 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:18:56.757 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.2 MB/s | 1.1 MB 00:00 2023-08-22T23:18:56.757 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:18:56.761 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:18:56.761 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:18:56.804 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:18:56.804 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:18:57.045 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:18:57.148 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:18:57.553 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:18:57.827 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:18:57.827 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:57.828 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:18:57.828 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:18:57.828 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:57.828 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:18:58.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw-debuginfo 2023-08-22T23:18:58.577 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:47 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:18:58.717 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:18:58.718 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 337 k 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:18:58.719 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:18:58.720 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:18:58.720 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:18:58.720 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 337 k 2023-08-22T23:18:58.720 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.0 M 2023-08-22T23:18:58.720 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:19:00.092 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rgw-debuginfo-18.0.0-5573.gf0ed7046.el9 246 kB/s | 337 kB 00:01 2023-08-22T23:19:00.093 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:00.094 INFO:teuthology.orchestra.run.smithi052.stdout:Total 245 kB/s | 337 kB 00:01 2023-08-22T23:19:00.094 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:19:00.098 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:19:00.099 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:19:00.141 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:19:00.141 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:19:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:19:00.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:19:00.784 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:19:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:19:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:01.071 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:19:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:01.071 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:19:01.291 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse-debuginfo 2023-08-22T23:19:01.824 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:50 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:19:01.961 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:19:01.962 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:01.962 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:19:01.962 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 506 k 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:19:01.963 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:01.964 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 506 k 2023-08-22T23:19:01.964 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.5 M 2023-08-22T23:19:01.964 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:19:02.387 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 1.2 MB/s | 506 kB 00:00 2023-08-22T23:19:02.388 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:02.389 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.2 MB/s | 506 kB 00:00 2023-08-22T23:19:02.389 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:19:02.392 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:19:02.392 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:19:02.430 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:19:02.430 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:19:02.599 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:19:02.705 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:19:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:19:03.412 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:19:03.412 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:03.413 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:19:03.413 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:03.413 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:03.413 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:19:03.646 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror-debuginfo 2023-08-22T23:19:04.176 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:53 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:19:04.317 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:19:04.317 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 51 M 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:19:04.318 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:04.319 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:19:04.319 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:04.319 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 51 M 2023-08-22T23:19:04.319 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 289 M 2023-08-22T23:19:04.319 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:19:07.051 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:19:08.174 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:19:08.582 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-debuginfo-18.0.0-5573.gf0ed7046.el9. 12 MB/s | 51 MB 00:04 2023-08-22T23:19:08.583 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:08.584 INFO:teuthology.orchestra.run.smithi052.stdout:Total 12 MB/s | 51 MB 00:04 2023-08-22T23:19:08.584 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:19:08.588 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:19:08.588 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:19:08.939 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:19:08.939 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:19:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:19:15.198 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:19:17.280 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:19:17.560 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:19:17.560 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:17.560 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:19:17.561 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:17.561 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:17.561 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:19:17.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd-debuginfo 2023-08-22T23:19:18.441 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:09:07 ago on Tue 22 Aug 2023 11:10:11 PM UTC. 2023-08-22T23:19:18.581 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2023-08-22T23:19:18.582 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:18.582 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2023-08-22T23:19:18.583 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:18.583 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2023-08-22T23:19:18.583 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.2 M 2023-08-22T23:19:18.583 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:18.583 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2023-08-22T23:19:18.583 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2023-08-22T23:19:18.584 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2023-08-22T23:19:18.584 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:18.584 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.2 M 2023-08-22T23:19:18.584 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 5.4 M 2023-08-22T23:19:18.585 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2023-08-22T23:19:19.260 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-debuginfo-18.0.0-5573.gf0ed7046.el9.x86 1.8 MB/s | 1.2 MB 00:00 2023-08-22T23:19:19.260 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:19.261 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.8 MB/s | 1.2 MB 00:00 2023-08-22T23:19:19.261 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2023-08-22T23:19:19.264 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2023-08-22T23:19:19.264 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2023-08-22T23:19:19.308 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2023-08-22T23:19:19.308 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2023-08-22T23:19:19.554 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2023-08-22T23:19:19.665 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:19:20.116 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:19:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:19:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:20.385 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2023-08-22T23:19:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:19:20.386 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2023-08-22T23:19:20.601 DEBUG:teuthology.parallel:result is None 2023-08-22T23:19:23.310 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:19:23.577 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:19:23.578 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:23.578 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:19:23.578 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:23.578 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:23.578 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:19:23.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse-debuginfo 2023-08-22T23:19:24.392 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:08 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:19:24.531 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:19:24.532 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:24.533 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:19:24.533 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:24.533 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:19:24.533 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 5.7 M 2023-08-22T23:19:24.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:24.534 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:19:24.534 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:24.534 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:19:24.534 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:24.535 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 5.7 M 2023-08-22T23:19:24.535 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 27 M 2023-08-22T23:19:24.535 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:19:25.197 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-fuse-debuginfo-18.0.0-5573.gf0ed7046.el9.x 8.6 MB/s | 5.7 MB 00:00 2023-08-22T23:19:25.197 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:25.198 INFO:teuthology.orchestra.run.smithi029.stdout:Total 8.6 MB/s | 5.7 MB 00:00 2023-08-22T23:19:25.198 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:19:25.202 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:19:25.202 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:19:25.273 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:19:25.274 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:19:26.035 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:19:26.138 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:19:27.007 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:19:27.285 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:19:27.286 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:27.286 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:19:27.286 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:27.286 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:27.286 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:19:27.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel-debuginfo 2023-08-22T23:19:28.089 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:12 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:19:28.232 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:19:28.233 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:28.233 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:19:28.233 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:28.233 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:19:28.233 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 109 k 2023-08-22T23:19:28.233 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 109 k 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 440 k 2023-08-22T23:19:28.234 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:19:28.564 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-debuginfo-18.0.0-5573.gf0ed7046. 328 kB/s | 109 kB 00:00 2023-08-22T23:19:28.565 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:28.565 INFO:teuthology.orchestra.run.smithi029.stdout:Total 327 kB/s | 109 kB 00:00 2023-08-22T23:19:28.565 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:19:28.569 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:19:28.569 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:19:28.606 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:19:28.606 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:19:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:19:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:19:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:19:29.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:19:29.643 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:29.647 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:19:29.647 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:29.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:29.647 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:19:29.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2-debuginfo 2023-08-22T23:19:30.432 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:14 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:19:30.569 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:19:30.570 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:30.570 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:19:30.570 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:30.570 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:19:30.571 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 4.9 M 2023-08-22T23:19:30.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:30.571 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:19:30.571 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:30.571 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:19:30.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:30.572 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 4.9 M 2023-08-22T23:19:30.572 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 23 M 2023-08-22T23:19:30.572 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:19:31.173 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs2-debuginfo-18.0.0-5573.gf0ed7046.el9. 8.2 MB/s | 4.9 MB 00:00 2023-08-22T23:19:31.173 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:31.174 INFO:teuthology.orchestra.run.smithi029.stdout:Total 8.1 MB/s | 4.9 MB 00:00 2023-08-22T23:19:31.174 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:19:31.177 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:19:31.178 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:19:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:19:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:19:31.902 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:19:32.007 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:19:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:19:33.018 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:19:33.018 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:33.018 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:19:33.018 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:33.019 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:33.019 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:19:33.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2-debuginfo 2023-08-22T23:19:33.811 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:17 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 44 M 2023-08-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 44 M 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 238 M 2023-08-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:19:37.054 INFO:teuthology.orchestra.run.smithi029.stdout:librados2-debuginfo-18.0.0-5573.gf0ed7046.el9.x 14 MB/s | 44 MB 00:03 2023-08-22T23:19:37.054 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:37.057 INFO:teuthology.orchestra.run.smithi029.stdout:Total 14 MB/s | 44 MB 00:03 2023-08-22T23:19:37.057 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:19:37.059 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:19:37.060 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:19:37.375 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:19:37.376 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:19:42.614 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:19:42.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:19:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:19:46.533 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_ 1/1 2023-08-22T23:19:46.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:46.534 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:19:46.534 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:19:46.534 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:46.535 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:19:46.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1-debuginfo 2023-08-22T23:19:47.345 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:31 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:19:47.487 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:19:47.488 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:47.488 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:19:47.488 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:47.488 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 51 M 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:19:47.489 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 51 M 2023-08-22T23:19:47.490 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 299 M 2023-08-22T23:19:47.490 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:19:58.462 INFO:teuthology.orchestra.run.smithi029.stdout:librbd1-debuginfo-18.0.0-5573.gf0ed7046.el9.x86 4.6 MB/s | 51 MB 00:10 2023-08-22T23:19:58.463 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:19:58.463 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.6 MB/s | 51 MB 00:10 2023-08-22T23:19:58.464 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:19:58.468 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:19:58.468 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:19:58.824 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:19:58.825 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:05.038 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:05.148 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:20:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:20:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:20:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:09.704 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:09.704 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:09.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs-debuginfo 2023-08-22T23:20:10.541 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:54 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:10.680 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:10.681 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:10.681 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 615 k 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:10.682 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:10.683 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 615 k 2023-08-22T23:20:10.683 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.1 M 2023-08-22T23:20:10.683 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:11.098 INFO:teuthology.orchestra.run.smithi029.stdout:python3-cephfs-debuginfo-18.0.0-5573.gf0ed7046. 1.4 MB/s | 615 kB 00:00 2023-08-22T23:20:11.099 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:11.099 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.4 MB/s | 615 kB 00:00 2023-08-22T23:20:11.099 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:11.102 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:11.102 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:11.142 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:11.143 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:11.314 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:11.474 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:20:11.968 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:20:12.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9 1/1 2023-08-22T23:20:12.279 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:12.279 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:12.279 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:12.279 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:12.279 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:12.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados-debuginfo 2023-08-22T23:20:13.105 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:57 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:13.247 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:13.248 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:13.248 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:13.248 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 769 k 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:13.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:13.250 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 769 k 2023-08-22T23:20:13.250 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.7 M 2023-08-22T23:20:13.250 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:13.696 INFO:teuthology.orchestra.run.smithi029.stdout:python3-rados-debuginfo-18.0.0-5573.gf0ed7046.e 1.7 MB/s | 769 kB 00:00 2023-08-22T23:20:13.697 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:13.697 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.7 MB/s | 769 kB 00:00 2023-08-22T23:20:13.697 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:13.701 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:13.701 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:13.741 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:13.742 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:13.931 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:14.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9. 1/1 2023-08-22T23:20:14.477 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9. 1/1 2023-08-22T23:20:14.756 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9. 1/1 2023-08-22T23:20:14.756 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:14.757 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:14.757 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:14.757 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:14.757 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:15.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd-debuginfo 2023-08-22T23:20:15.563 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:09:59 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:15.702 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:15.703 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:15.703 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:15.704 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:15.704 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:15.704 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.1 M 2023-08-22T23:20:15.704 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:15.704 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:15.704 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:15.705 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:15.705 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:15.705 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.1 M 2023-08-22T23:20:15.705 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 4.3 M 2023-08-22T23:20:15.705 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:16.021 INFO:teuthology.orchestra.run.smithi029.stdout:python3-rbd-debuginfo-18.0.0-5573.gf0ed7046.el9 3.5 MB/s | 1.1 MB 00:00 2023-08-22T23:20:16.022 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:16.022 INFO:teuthology.orchestra.run.smithi029.stdout:Total 3.5 MB/s | 1.1 MB 00:00 2023-08-22T23:20:16.022 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:16.026 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:16.026 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:16.068 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:16.068 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:16.295 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:16.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:20:16.942 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:20:17.252 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:20:17.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:17.253 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:17.253 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:17.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:17.253 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:17.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw-debuginfo 2023-08-22T23:20:18.073 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:10:02 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:18.212 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:18.213 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:18.213 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:18.213 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 337 k 2023-08-22T23:20:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 337 k 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.0 M 2023-08-22T23:20:18.214 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:18.471 INFO:teuthology.orchestra.run.smithi029.stdout:python3-rgw-debuginfo-18.0.0-5573.gf0ed7046.el9 1.3 MB/s | 337 kB 00:00 2023-08-22T23:20:18.471 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:18.472 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.3 MB/s | 337 kB 00:00 2023-08-22T23:20:18.472 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:18.523 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:18.523 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:18.668 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:18.813 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:20:19.213 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:20:19.527 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x8 1/1 2023-08-22T23:20:19.527 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:19.528 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:19.528 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:19.528 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:19.528 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:19.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse-debuginfo 2023-08-22T23:20:20.354 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:10:04 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:20.493 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:20.494 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:20.494 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:20.494 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 506 k 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:20.495 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:20.496 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 506 k 2023-08-22T23:20:20.496 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.5 M 2023-08-22T23:20:20.496 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:20.904 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-debuginfo-18.0.0-5573.gf0ed7046.el9.x8 1.2 MB/s | 506 kB 00:00 2023-08-22T23:20:20.905 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:20.905 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.2 MB/s | 506 kB 00:00 2023-08-22T23:20:20.905 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:20.909 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:20.909 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:20.948 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:20.948 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:21.117 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:21.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:20:21.771 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:20:22.079 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_6 1/1 2023-08-22T23:20:22.079 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:22.080 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:22.080 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:22.080 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:22.080 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:22.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror-debuginfo 2023-08-22T23:20:22.898 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:10:06 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:23.037 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:23.038 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:23.038 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:23.038 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:23.038 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:23.038 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 51 M 2023-08-22T23:20:23.038 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 51 M 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 289 M 2023-08-22T23:20:23.039 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:27.721 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-debuginfo-18.0.0-5573.gf0ed7046.el9. 11 MB/s | 51 MB 00:04 2023-08-22T23:20:27.722 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:27.722 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 51 MB 00:04 2023-08-22T23:20:27.723 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:27.727 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:27.727 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:28.078 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:28.079 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:34.234 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:34.346 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:20:38.487 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:20:38.776 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86 1/1 2023-08-22T23:20:38.776 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:38.776 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:38.776 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:38.777 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:38.777 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:39.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd-debuginfo 2023-08-22T23:20:39.589 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:10:23 ago on Tue 22 Aug 2023 11:10:16 PM UTC. 2023-08-22T23:20:39.730 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-08-22T23:20:39.730 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:39.731 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-08-22T23:20:39.731 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:39.731 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-08-22T23:20:39.731 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-5573.gf0ed7046.el9 ceph 1.2 M 2023-08-22T23:20:39.731 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:39.731 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-08-22T23:20:39.732 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-08-22T23:20:39.732 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-08-22T23:20:39.732 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:39.732 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.2 M 2023-08-22T23:20:39.732 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 5.4 M 2023-08-22T23:20:39.732 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-08-22T23:20:40.186 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-debuginfo-18.0.0-5573.gf0ed7046.el9.x86 2.7 MB/s | 1.2 MB 00:00 2023-08-22T23:20:40.187 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-08-22T23:20:40.187 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.7 MB/s | 1.2 MB 00:00 2023-08-22T23:20:40.187 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-08-22T23:20:40.191 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-08-22T23:20:40.191 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-08-22T23:20:40.235 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-08-22T23:20:40.235 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-08-22T23:20:40.478 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-08-22T23:20:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:20:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:20:41.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 1/1 2023-08-22T23:20:41.462 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:41.462 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-08-22T23:20:41.462 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-debuginfo-2:18.0.0-5573.gf0ed7046.el9.x86_64 2023-08-22T23:20:41.462 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:41.462 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-08-22T23:20:41.741 DEBUG:teuthology.parallel:result is None 2023-08-22T23:20:41.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T23:20:41.890 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-22T23:20:41.910 INFO:teuthology.orchestra.run.smithi029.stdout:18.0.0-5573.gf0ed7046.el9 2023-08-22T23:20:41.911 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5573.gf0ed7046.el9 2023-08-22T23:20:41.912 INFO:teuthology.task.install:The correct ceph version 18.0.0-5573.gf0ed7046 is installed. 2023-08-22T23:20:41.912 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T23:20:42.052 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-22T23:20:42.078 INFO:teuthology.orchestra.run.smithi052.stdout:18.0.0-5573.gf0ed7046.el9 2023-08-22T23:20:42.078 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5573.gf0ed7046.el9 2023-08-22T23:20:42.079 INFO:teuthology.task.install:The correct ceph version 18.0.0-5573.gf0ed7046 is installed. 2023-08-22T23:20:42.079 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-08-22T23:20:42.080 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:42.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-22T23:20:42.112 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:42.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-22T23:20:42.155 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-08-22T23:20:42.156 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:42.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-08-22T23:20:42.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-22T23:20:42.264 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:42.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2023-08-22T23:20:42.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-22T23:20:42.379 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-08-22T23:20:42.380 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:42.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-22T23:20:42.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-22T23:20:42.520 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:42.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-22T23:20:42.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-22T23:20:42.637 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-08-22T23:20:42.638 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:42.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-08-22T23:20:42.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-22T23:20:42.753 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:42.754 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2023-08-22T23:20:42.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-22T23:20:42.865 INFO:teuthology.run_tasks:Running task ceph... 2023-08-22T23:20:43.139 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-08-22T23:20:43.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2023-08-22T23:20:43.147 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2023-08-22T23:20:43.183 INFO:tasks.ceph:Disabling ceph logrotate... 2023-08-22T23:20:43.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2023-08-22T23:20:43.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2023-08-22T23:20:43.263 INFO:tasks.ceph:Creating extra log directories... 2023-08-22T23:20:43.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-08-22T23:20:43.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-08-22T23:20:43.350 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-08-22T23:20:43.351 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-08-22T23:20:43.351 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793', 'branch': 'wip-yuri10-testing-2023-08-17-1444', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7376793', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2023-08-17-1444'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2023-08-17-1444', 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4042, 'sha1': 'f0ed704638a5702969ce59fb4418946a60474648', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri10-testing-2023-08-17-1444', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f0ed704638a5702969ce59fb4418946a60474648', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+3T+PSpbH3t890q4p+Mct7FKdX102eb77eXtY6aQVYryOmE5LO4b2qE6gLs6SdX09JTZKaLeLIpqbWMKtb1Vc=', 'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG7NfRtW1+ZXWRHpP4aldS+1SQUMyv292c+czbs0gFMRfifl6nyklgZeVwlja+SpA4PHEyREN0RAP93I5k6s4eo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '449a1bc2027504e7b3c3d7b30fa4178906581da7', 'timestamp': '2023-08-22_18:16:03', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2224'} 2023-08-22T23:20:43.352 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-08-22T23:20:43.368 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-08-22T23:20:43.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-08-22T23:20:43.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2023-08-22T23:20:43.479 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:43.480 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2023-08-22T23:20:43.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-22T23:20:43.513 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2023-08-22T23:20:43.570 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-22T23:20:43.571 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:43.571 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 656 Links: 1 2023-08-22T23:20:43.571 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:43.571 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:43.571 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-08-22 23:12:18.837821975 +0000 2023-08-22T23:20:43.572 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-08-22 23:09:20.914061284 +0000 2023-08-22T23:20:43.572 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-08-22 23:09:20.914061284 +0000 2023-08-22T23:20:43.572 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-08-22 23:09:20.914061284 +0000 2023-08-22T23:20:43.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-22T23:20:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-08-22T23:20:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-08-22T23:20:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.8647e-05 s, 5.2 MB/s 2023-08-22T23:20:43.644 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-22T23:20:43.702 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2023-08-22T23:20:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-22T23:20:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 672 Links: 1 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-08-22 23:12:18.837821975 +0000 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-08-22 23:09:21.450052613 +0000 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-08-22 23:09:21.450052613 +0000 2023-08-22T23:20:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-08-22 23:09:21.450052613 +0000 2023-08-22T23:20:43.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-22T23:20:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-08-22T23:20:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-08-22T23:20:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.643e-05 s, 5.3 MB/s 2023-08-22T23:20:43.832 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-22T23:20:43.889 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2023-08-22T23:20:43.948 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-22T23:20:43.949 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:43.949 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 690 Links: 1 2023-08-22T23:20:43.950 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:43.950 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:43.951 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-08-22 23:12:18.837821975 +0000 2023-08-22T23:20:43.952 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-08-22 23:09:21.969044217 +0000 2023-08-22T23:20:43.953 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-08-22 23:09:21.969044217 +0000 2023-08-22T23:20:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-08-22 23:09:21.969044217 +0000 2023-08-22T23:20:43.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-22T23:20:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-08-22T23:20:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-08-22T23:20:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.1757e-05 s, 6.3 MB/s 2023-08-22T23:20:44.018 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-22T23:20:44.074 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2023-08-22T23:20:44.129 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-22T23:20:44.129 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:44.129 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 709 Links: 1 2023-08-22T23:20:44.130 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:44.130 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:44.130 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-08-22 23:12:18.838821959 +0000 2023-08-22T23:20:44.130 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-08-22 23:09:22.485035870 +0000 2023-08-22T23:20:44.130 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-08-22 23:09:22.485035870 +0000 2023-08-22T23:20:44.130 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-08-22 23:09:22.485035870 +0000 2023-08-22T23:20:44.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-22T23:20:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-08-22T23:20:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-08-22T23:20:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.0533e-05 s, 5.7 MB/s 2023-08-22T23:20:44.199 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-22T23:20:44.255 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:44.256 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:44.256 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2023-08-22T23:20:44.274 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-22T23:20:44.275 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2023-08-22T23:20:44.333 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-22T23:20:44.333 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:44.333 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 656 Links: 1 2023-08-22T23:20:44.333 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:44.333 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:44.334 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-08-22 23:11:58.376287487 +0000 2023-08-22T23:20:44.334 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-08-22 23:09:16.490318261 +0000 2023-08-22T23:20:44.334 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-08-22 23:09:16.490318261 +0000 2023-08-22T23:20:44.334 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-08-22 23:09:16.490318261 +0000 2023-08-22T23:20:44.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-22T23:20:44.407 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-08-22T23:20:44.407 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-08-22T23:20:44.407 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.7667e-05 s, 5.2 MB/s 2023-08-22T23:20:44.408 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-22T23:20:44.466 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2023-08-22T23:20:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 674 Links: 1 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-08-22 23:11:58.376287487 +0000 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-08-22 23:09:17.008311683 +0000 2023-08-22T23:20:44.524 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-08-22 23:09:17.008311683 +0000 2023-08-22T23:20:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-08-22 23:09:17.008311683 +0000 2023-08-22T23:20:44.525 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-22T23:20:44.598 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-08-22T23:20:44.598 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-08-22T23:20:44.599 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.9245e-05 s, 5.2 MB/s 2023-08-22T23:20:44.600 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-22T23:20:44.658 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2023-08-22T23:20:44.715 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-22T23:20:44.715 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-08-22 23:11:58.377287475 +0000 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-08-22 23:09:17.504305385 +0000 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-08-22 23:09:17.504305385 +0000 2023-08-22T23:20:44.716 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-08-22 23:09:17.504305385 +0000 2023-08-22T23:20:44.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-22T23:20:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-08-22T23:20:44.788 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-08-22T23:20:44.788 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000110105 s, 4.7 MB/s 2023-08-22T23:20:44.789 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-22T23:20:44.846 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2023-08-22T23:20:44.903 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-22T23:20:44.904 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-22T23:20:44.904 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 710 Links: 1 2023-08-22T23:20:44.904 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-22T23:20:44.904 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2023-08-22T23:20:44.904 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2023-08-22 23:11:58.377287475 +0000 2023-08-22T23:20:44.905 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2023-08-22 23:09:18.041298569 +0000 2023-08-22T23:20:44.905 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2023-08-22 23:09:18.041298569 +0000 2023-08-22T23:20:44.905 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2023-08-22 23:09:18.041298569 +0000 2023-08-22T23:20:44.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-22T23:20:44.975 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2023-08-22T23:20:44.976 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2023-08-22T23:20:44.976 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.5476e-05 s, 6.0 MB/s 2023-08-22T23:20:44.977 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-22T23:20:45.033 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:45.034 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-08-22T23:20:45.035 INFO:tasks.ceph:Generating config... 2023-08-22T23:20:45.038 INFO:tasks.ceph:[client] debug monc = 20 2023-08-22T23:20:45.039 INFO:tasks.ceph:[client] debug ms = 1 2023-08-22T23:20:45.039 INFO:tasks.ceph:[client] debug objecter = 20 2023-08-22T23:20:45.040 INFO:tasks.ceph:[client] debug rados = 20 2023-08-22T23:20:45.040 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-08-22T23:20:45.041 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-08-22T23:20:45.041 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-08-22T23:20:45.041 INFO:tasks.ceph:[global] ms type = async 2023-08-22T23:20:45.042 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-08-22T23:20:45.042 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-08-22T23:20:45.042 INFO:tasks.ceph:[mgr] debug ms = 1 2023-08-22T23:20:45.043 INFO:tasks.ceph:[mon] debug mon = 20 2023-08-22T23:20:45.043 INFO:tasks.ceph:[mon] debug ms = 1 2023-08-22T23:20:45.043 INFO:tasks.ceph:[mon] debug paxos = 20 2023-08-22T23:20:45.044 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-08-22T23:20:45.044 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-08-22T23:20:45.045 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-08-22T23:20:45.045 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-08-22T23:20:45.045 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-08-22T23:20:45.045 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-08-22T23:20:45.046 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-08-22T23:20:45.046 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-08-22T23:20:45.046 INFO:tasks.ceph:[osd] bdev async discard = True 2023-08-22T23:20:45.046 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-08-22T23:20:45.047 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-08-22T23:20:45.047 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-08-22T23:20:45.047 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-08-22T23:20:45.047 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-08-22T23:20:45.047 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-08-22T23:20:45.048 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-08-22T23:20:45.048 INFO:tasks.ceph:[osd] debug monc = 20 2023-08-22T23:20:45.048 INFO:tasks.ceph:[osd] debug ms = 1 2023-08-22T23:20:45.048 INFO:tasks.ceph:[osd] debug osd = 20 2023-08-22T23:20:45.048 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-08-22T23:20:45.049 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-08-22T23:20:45.049 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-08-22T23:20:45.049 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-08-22T23:20:45.049 INFO:tasks.ceph:[osd] osd class default list = * 2023-08-22T23:20:45.049 INFO:tasks.ceph:[osd] osd class load list = * 2023-08-22T23:20:45.050 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-08-22T23:20:45.050 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-08-22T23:20:45.050 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-08-22T23:20:45.050 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-08-22T23:20:45.050 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-08-22T23:20:45.050 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-08-22T23:20:45.051 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-08-22T23:20:45.051 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-08-22T23:20:45.051 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-08-22T23:20:45.051 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-08-22T23:20:45.051 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-08-22T23:20:45.052 INFO:tasks.ceph:Setting up mon.a... 2023-08-22T23:20:45.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-08-22T23:20:45.115 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2023-08-22T23:20:45.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-08-22T23:20:45.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-08-22T23:20:45.198 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29'), ('mon.c', '[v2:172.21.15.29:3301,v1:172.21.15.29:6790]'), ('mon.b', '172.21.15.52')] 2023-08-22T23:20:45.199 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29,[v2:172.21.15.29:3301,v1:172.21.15.29:6790],172.21.15.52', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-08-22T23:20:45.200 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:45.201 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-08-22T23:20:45.254 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --addv c '[v2:172.21.15.29:3301,v1:172.21.15.29:6790]' --add b 172.21.15.52 --print /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:45.340 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:45.340 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:20:45.340 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = pacific 2023-08-22T23:20:45.340 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 16 (pacific) 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:20:45.341 INFO:teuthology.orchestra.run.smithi029.stdout:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:20:45.342 INFO:teuthology.orchestra.run.smithi029.stdout:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:20:45.342 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-08-22T23:20:45.343 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-08-22T23:20:45.358 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cfd0f96d-1c62-4667-a308-870e3273ade2... 2023-08-22T23:20:45.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-08-22T23:20:45.400 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-08-22T23:20:45.458 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-08-22T23:20:45.458 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-08-22T23:20:45.459 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-22T23:20:45.459 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-08-22T23:20:45.459 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.459 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-08-22T23:20:45.459 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.460 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-08-22T23:20:45.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.460 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-08-22T23:20:45.460 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-08-22T23:20:45.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.461 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-08-22T23:20:45.461 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-08-22T23:20:45.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.462 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-22T23:20:45.462 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-08-22T23:20:45.462 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.462 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-08-22T23:20:45.463 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-22T23:20:45.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.463 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-08-22T23:20:45.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.464 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-08-22T23:20:45.464 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-08-22T23:20:45.464 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-08-22T23:20:45.464 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-08-22T23:20:45.465 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-08-22T23:20:45.466 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "172.21.15.29,[v2:172.21.15.29:3301,v1:172.21.15.29:6790],172.21.15.52" 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: mon election default strategy = 1 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 5000 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: ms type = async 2023-08-22T23:20:45.467 INFO:teuthology.orchestra.run.smithi029.stdout: osd heartbeat grace = 80 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-08-22T23:20:45.468 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-08-22T23:20:45.469 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-08-22T23:20:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: bdev async discard = True 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: bdev enable discard = True 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = bitmap 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore zero block detection = True 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1 2023-08-22T23:20:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: osd class default list = * 2023-08-22T23:20:45.473 INFO:teuthology.orchestra.run.smithi029.stdout: osd class load list = * 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd client watch timeout = 120 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify cached snaps = True 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify missing on start = True 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd fast shutdown = False 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock override recovery settings = True 2023-08-22T23:20:45.474 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue = debug_random 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue cut off = debug_random 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: osd shutdown pgref assert = True 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-08-22T23:20:45.475 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-08-22T23:20:45.476 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-08-22T23:20:45.477 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-08-22T23:20:45.478 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-08-22T23:20:45.478 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.478 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-22T23:20:45.478 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-08-22T23:20:45.478 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-08-22T23:20:45.479 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon min osdmap epochs = 50 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd crush smoke test = False 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune interval = 2 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune min = 15 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune txsize = 2 2023-08-22T23:20:45.480 INFO:teuthology.orchestra.run.smithi029.stdout: mon scrub interval = 300 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on pool no app = False 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout: paxos service trim min = 10 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-08-22T23:20:45.481 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout: debug objecter = 20 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout: debug rados = 20 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-08-22T23:20:45.482 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2023-08-22T23:20:45.483 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2023-08-22T23:20:45.483 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2023-08-22T23:20:45.483 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2023-08-22T23:20:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-22T23:20:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2023-08-22T23:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-22T23:20:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2023-08-22T23:20:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2023-08-22T23:20:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2023-08-22T23:20:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2023-08-22T23:20:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = "172.21.15.29,[v2:172.21.15.29:3301,v1:172.21.15.29:6790],172.21.15.52" 2023-08-22T23:20:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: mon client directed command retry = 5 2023-08-22T23:20:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: mon election default strategy = 1 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: ms inject socket failures = 5000 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: ms type = async 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: osd heartbeat grace = 80 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2023-08-22T23:20:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2023-08-22T23:20:45.491 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.492 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2023-08-22T23:20:45.493 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2023-08-22T23:20:45.494 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.494 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2023-08-22T23:20:45.494 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2023-08-22T23:20:45.494 INFO:teuthology.orchestra.run.smithi052.stdout: bdev async discard = True 2023-08-22T23:20:45.495 INFO:teuthology.orchestra.run.smithi052.stdout: bdev enable discard = True 2023-08-22T23:20:45.495 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore allocator = bitmap 2023-08-22T23:20:45.495 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2023-08-22T23:20:45.495 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore zero block detection = True 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 1 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 1 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 10 2023-08-22T23:20:45.496 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: osd class default list = * 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: osd class load list = * 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: osd client watch timeout = 120 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify cached snaps = True 2023-08-22T23:20:45.497 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify missing on start = True 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd fast shutdown = False 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock override recovery settings = True 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue = debug_random 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue cut off = debug_random 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: osd shutdown pgref assert = True 2023-08-22T23:20:45.498 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2023-08-22T23:20:45.499 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2023-08-22T23:20:45.500 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2023-08-22T23:20:45.501 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2023-08-22T23:20:45.502 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon min osdmap epochs = 50 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd crush smoke test = False 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune interval = 2 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune min = 15 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune txsize = 2 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon scrub interval = 300 2023-08-22T23:20:45.503 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on pool no app = False 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout: paxos service trim min = 10 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2023-08-22T23:20:45.504 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout: debug objecter = 20 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout: debug rados = 20 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.c] 2023-08-22T23:20:45.505 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.b] 2023-08-22T23:20:45.507 INFO:tasks.ceph:Creating admin key on mon.a... 2023-08-22T23:20:45.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-08-22T23:20:45.567 INFO:tasks.ceph:Copying monmap to all nodes... 2023-08-22T23:20:45.567 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:45.567 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-08-22T23:20:45.583 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:45.583 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-08-22T23:20:45.637 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:20:45.637 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:45.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2023-08-22T23:20:45.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-08-22T23:20:45.724 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:20:45.724 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:45.778 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:20:45.778 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:45.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.keyring 2023-08-22T23:20:45.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-08-22T23:20:45.830 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:20:45.830 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:45.886 INFO:tasks.ceph:Setting up mon nodes... 2023-08-22T23:20:45.886 INFO:tasks.ceph:Setting up mgr nodes... 2023-08-22T23:20:45.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-08-22T23:20:45.950 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-08-22T23:20:45.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-08-22T23:20:46.021 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-08-22T23:20:46.025 INFO:tasks.ceph:Setting up mds nodes... 2023-08-22T23:20:46.025 INFO:tasks.ceph_client:Setting up client nodes... 2023-08-22T23:20:46.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-08-22T23:20:46.073 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-08-22T23:20:46.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-08-22T23:20:46.139 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-08-22T23:20:46.157 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-08-22T23:20:46.158 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-08-22T23:20:46.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-08-22T23:20:46.189 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:46.190 INFO:tasks.ceph:role: osd.0 2023-08-22T23:20:46.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:20:46.190 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-08-22T23:20:46.259 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:20:46.260 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:20:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:20:46.261 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:20:46.283 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-08-22T23:20:46.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-08-22T23:20:46.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-08-22T23:20:46.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-08-22T23:20:46.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-08-22T23:20:46.506 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:46.506 INFO:tasks.ceph:role: osd.1 2023-08-22T23:20:46.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:20:46.507 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-08-22T23:20:46.576 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:20:46.576 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:20:46.576 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:20:46.576 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:20:46.577 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:20:46.577 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:20:46.577 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:20:46.577 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:20:46.577 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:20:46.577 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:20:46.599 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-08-22T23:20:46.601 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-08-22T23:20:46.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-08-22T23:20:46.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-08-22T23:20:46.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-08-22T23:20:46.819 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:46.819 INFO:tasks.ceph:role: osd.2 2023-08-22T23:20:46.819 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:20:46.819 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-08-22T23:20:46.889 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:20:46.889 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:20:46.889 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:20:46.889 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:20:46.889 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:20:46.890 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:20:46.890 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:20:46.890 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:20:46.890 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:20:46.890 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:20:46.913 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-08-22T23:20:46.914 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-08-22T23:20:46.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-08-22T23:20:46.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-08-22T23:20:47.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-08-22T23:20:47.133 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:47.134 INFO:tasks.ceph:role: osd.3 2023-08-22T23:20:47.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:20:47.135 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-08-22T23:20:47.203 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:20:47.203 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:20:47.203 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:20:47.203 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:20:47.204 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:20:47.204 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:20:47.204 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:20:47.204 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:20:47.204 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:20:47.204 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:20:47.227 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-08-22T23:20:47.229 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-08-22T23:20:47.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-08-22T23:20:47.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-08-22T23:20:47.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:47.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:47.479+0000 7f872ea0f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-08-22T23:20:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:47.479+0000 7f872ea0f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-08-22T23:20:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:47.479+0000 7f872ea0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-08-22T23:20:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:47.479+0000 7f872ea0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-08-22T23:20:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:47.479+0000 7f872ea0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-08-22T23:20:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:47.480+0000 7f872ea0f500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-08-22T23:20:50.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-08-22T23:20:50.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:50.456+0000 7fa26eb82500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-08-22T23:20:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:50.456+0000 7fa26eb82500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-08-22T23:20:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:50.456+0000 7fa26eb82500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-08-22T23:20:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:50.456+0000 7fa26eb82500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-08-22T23:20:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:50.456+0000 7fa26eb82500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-08-22T23:20:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:50.457+0000 7fa26eb82500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-08-22T23:20:53.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-08-22T23:20:53.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:53.450+0000 7f78d1a8a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-08-22T23:20:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:53.450+0000 7f78d1a8a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-08-22T23:20:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:53.450+0000 7f78d1a8a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-08-22T23:20:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:53.450+0000 7f78d1a8a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-08-22T23:20:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:53.450+0000 7f78d1a8a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-08-22T23:20:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:53.450+0000 7f78d1a8a500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-08-22T23:20:56.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-08-22T23:20:56.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:20:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:56.418+0000 7f0f994d0500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-08-22T23:20:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:56.418+0000 7f0f994d0500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-08-22T23:20:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:56.418+0000 7f0f994d0500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-08-22T23:20:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:56.418+0000 7f0f994d0500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-08-22T23:20:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:56.418+0000 7f0f994d0500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-08-22T23:20:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:20:56.418+0000 7f0f994d0500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-08-22T23:20:59.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-08-22T23:20:59.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-08-22T23:20:59.336 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:59.337 INFO:tasks.ceph:role: osd.4 2023-08-22T23:20:59.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:20:59.337 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-08-22T23:20:59.410 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:20:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:20:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:20:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:20:59.411 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:20:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:20:59.411 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:20:59.411 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:20:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:20:59.411 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:20:59.434 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2023-08-22T23:20:59.436 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2023-08-22T23:20:59.436 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-08-22T23:20:59.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-08-22T23:20:59.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-08-22T23:20:59.660 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:59.660 INFO:tasks.ceph:role: osd.5 2023-08-22T23:20:59.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:20:59.660 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-08-22T23:20:59.730 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:20:59.730 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:20:59.730 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:20:59.731 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:20:59.753 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2023-08-22T23:20:59.755 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2023-08-22T23:20:59.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-08-22T23:20:59.829 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-08-22T23:20:59.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-08-22T23:20:59.973 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-22T23:20:59.974 INFO:tasks.ceph:role: osd.6 2023-08-22T23:20:59.974 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:20:59.974 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-08-22T23:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:21:00.069 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2023-08-22T23:21:00.070 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2023-08-22T23:21:00.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-08-22T23:21:00.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-08-22T23:21:00.218 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-08-22T23:21:00.288 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-22T23:21:00.288 INFO:tasks.ceph:role: osd.7 2023-08-22T23:21:00.289 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:21:00.289 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-08-22T23:21:00.357 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-08-22T23:21:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-22T23:21:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-22T23:21:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-08-22T23:21:00.358 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-22T23:21:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2023-08-22T23:21:00.358 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-22T23:21:00.359 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-22T23:21:00.359 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-22T23:21:00.359 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-22T23:21:00.381 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2023-08-22T23:21:00.383 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2023-08-22T23:21:00.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-08-22T23:21:00.457 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-08-22T23:21:00.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:21:00.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:00.632+0000 7f1d870f9500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-08-22T23:21:00.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:00.632+0000 7f1d870f9500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-08-22T23:21:00.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:00.632+0000 7f1d870f9500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-08-22T23:21:00.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:00.633+0000 7f1d870f9500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-08-22T23:21:00.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:00.633+0000 7f1d870f9500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-08-22T23:21:00.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:00.633+0000 7f1d870f9500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-08-22T23:21:03.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-08-22T23:21:03.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:21:03.626 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:03.624+0000 7f6eb97bc500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-08-22T23:21:03.626 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:03.624+0000 7f6eb97bc500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-08-22T23:21:03.626 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:03.624+0000 7f6eb97bc500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-08-22T23:21:03.627 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:03.624+0000 7f6eb97bc500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-08-22T23:21:03.627 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:03.624+0000 7f6eb97bc500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-08-22T23:21:03.627 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:03.625+0000 7f6eb97bc500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-08-22T23:21:06.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-08-22T23:21:06.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:21:06.596 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:06.594+0000 7fe573b82500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-08-22T23:21:06.596 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:06.595+0000 7fe573b82500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-08-22T23:21:06.596 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:06.595+0000 7fe573b82500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-08-22T23:21:06.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:06.595+0000 7fe573b82500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-08-22T23:21:06.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:06.595+0000 7fe573b82500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-08-22T23:21:06.597 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:06.595+0000 7fe573b82500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-08-22T23:21:09.436 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-08-22T23:21:09.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:21:09.566 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:09.565+0000 7f2b47214500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-08-22T23:21:09.567 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:09.565+0000 7f2b47214500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-08-22T23:21:09.567 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:09.565+0000 7f2b47214500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-08-22T23:21:09.567 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:09.565+0000 7f2b47214500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-08-22T23:21:09.567 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:09.565+0000 7f2b47214500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-08-22T23:21:09.567 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:09.565+0000 7f2b47214500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-08-22T23:21:12.407 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-08-22T23:21:12.442 INFO:tasks.ceph:Reading keys from all nodes... 2023-08-22T23:21:12.443 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:12.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-08-22T23:21:12.479 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:12.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-08-22T23:21:12.553 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:12.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-08-22T23:21:12.626 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:12.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-08-22T23:21:12.699 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:12.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-08-22T23:21:12.772 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:12.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-08-22T23:21:12.808 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:12.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-08-22T23:21:12.882 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:12.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-08-22T23:21:12.955 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:12.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-08-22T23:21:13.026 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:13.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-08-22T23:21:13.097 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:13.097 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-08-22T23:21:13.114 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:13.115 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-08-22T23:21:13.154 INFO:tasks.ceph:Adding keys to all mons... 2023-08-22T23:21:13.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2023-08-22T23:21:13.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/ceph.keyring 2023-08-22T23:21:13.199 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2023-08-22T23:21:13.199 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDNQuVkaC6nOBAA9wFKwEUpgEGpn0YUsogC2g== 2023-08-22T23:21:13.199 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2023-08-22T23:21:13.199 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDPQuVk/iesHBAAc7ih3RSHn5eHg4Mhyqz/GQ== 2023-08-22T23:21:13.199 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDSQuVkBSxJGxAAduy5bzcOmF9AhYQshLnJLA== 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDVQuVkDY/sGhAA5EVVX5zGvGdMjZ2drrNJCw== 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDYQuVk+p0AGRAANmaOg+ehsdLS93xMEKsXbQ== 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2023-08-22T23:21:13.200 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDOQuVk7bRJARAAnRuEZV3NnqlzqHlKaakW2w== 2023-08-22T23:21:13.201 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2023-08-22T23:21:13.201 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDcQuVkh4HHJRAA8EvAClfYGQOy8nFUvBHpUw== 2023-08-22T23:21:13.201 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2023-08-22T23:21:13.201 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDfQuVkSE1MJRAAYpEkELnjnDXuA5sjdVdDzw== 2023-08-22T23:21:13.201 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2023-08-22T23:21:13.202 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDiQuVkfs2EIxAAoGLH5jYfnFywEOK5EjooCg== 2023-08-22T23:21:13.202 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2023-08-22T23:21:13.202 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDlQuVk9YHBIRAAMVTAkbPBIzJAkUh/kkwh2w== 2023-08-22T23:21:13.202 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-08-22T23:21:13.202 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDOQuVkKNVcBBAA+W+euJHB3wGZdVzQW05cRQ== 2023-08-22T23:21:13.202 INFO:teuthology.orchestra.run.smithi029.stdout:[client.1] 2023-08-22T23:21:13.203 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDOQuVkNNNMCBAAG2kTC1CkvOyZUf2Fkk/YBA== 2023-08-22T23:21:13.225 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.y] 2023-08-22T23:21:13.225 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDNQuVkaC6nOBAA9wFKwEUpgEGpn0YUsogC2g== 2023-08-22T23:21:13.225 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2023-08-22T23:21:13.225 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDPQuVk/iesHBAAc7ih3RSHn5eHg4Mhyqz/GQ== 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDSQuVkBSxJGxAAduy5bzcOmF9AhYQshLnJLA== 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDVQuVkDY/sGhAA5EVVX5zGvGdMjZ2drrNJCw== 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.3] 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDYQuVk+p0AGRAANmaOg+ehsdLS93xMEKsXbQ== 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2023-08-22T23:21:13.226 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDOQuVk7bRJARAAnRuEZV3NnqlzqHlKaakW2w== 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.4] 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDcQuVkh4HHJRAA8EvAClfYGQOy8nFUvBHpUw== 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.5] 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDfQuVkSE1MJRAAYpEkELnjnDXuA5sjdVdDzw== 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.6] 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDiQuVkfs2EIxAAoGLH5jYfnFywEOK5EjooCg== 2023-08-22T23:21:13.227 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.7] 2023-08-22T23:21:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDlQuVk9YHBIRAAMVTAkbPBIzJAkUh/kkwh2w== 2023-08-22T23:21:13.228 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2023-08-22T23:21:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDOQuVkKNVcBBAA+W+euJHB3wGZdVzQW05cRQ== 2023-08-22T23:21:13.228 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2023-08-22T23:21:13.228 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDOQuVkNNNMCBAAG2kTC1CkvOyZUf2Fkk/YBA== 2023-08-22T23:21:13.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-22T23:21:13.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-22T23:21:13.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.385 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.443 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-22T23:21:13.559 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-22T23:21:13.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-22T23:21:13.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-22T23:21:13.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-22T23:21:13.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-22T23:21:13.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-22T23:21:13.963 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-08-22T23:21:13.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-08-22T23:21:13.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-08-22T23:21:14.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-08-22T23:21:14.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-08-22T23:21:14.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-08-22T23:21:14.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-08-22T23:21:14.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-08-22T23:21:14.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-08-22T23:21:14.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-08-22T23:21:14.493 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:21:14.496 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-08-22T23:21:14.548 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-08-22T23:21:14.549 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-08-22T23:21:14.549 INFO:tasks.ceph.mon.a:Restarting daemon 2023-08-22T23:21:14.550 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-08-22T23:21:14.552 INFO:tasks.ceph.mon.a:Started 2023-08-22T23:21:14.552 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-08-22T23:21:14.552 INFO:tasks.ceph.mon.c:Restarting daemon 2023-08-22T23:21:14.553 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-08-22T23:21:14.554 INFO:tasks.ceph.mon.c:Started 2023-08-22T23:21:14.555 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-08-22T23:21:14.555 INFO:tasks.ceph.mon.b:Restarting daemon 2023-08-22T23:21:14.556 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-08-22T23:21:14.592 INFO:tasks.ceph.mon.b:Started 2023-08-22T23:21:14.592 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-08-22T23:21:14.593 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-08-22T23:21:14.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-08-22T23:21:14.595 INFO:tasks.ceph.mgr.y:Started 2023-08-22T23:21:14.596 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-08-22T23:21:14.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-08-22T23:21:14.598 INFO:tasks.ceph.mgr.x:Started 2023-08-22T23:21:14.599 INFO:tasks.ceph:Setting crush tunables to default 2023-08-22T23:21:14.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2023-08-22T23:21:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.703+0000 7f4dca6b5640 10 monclient: get_monmap_and_config 2023-08-22T23:21:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.703+0000 7f4dca6b5640 10 monclient: build_initial_monmap 2023-08-22T23:21:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.703+0000 7f4dca6b5640 10 monclient: monmap: 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:21:14.704238+0000 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:21:14.704238+0000 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:14.706 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.704+0000 7f4dca6b5640 1 Processor -- start 2023-08-22T23:21:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.704+0000 7f4dca6b5640 1 -- start start 2023-08-22T23:21:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.704+0000 7f4dca6b5640 10 monclient: init 2023-08-22T23:21:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-08-22T23:21:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 0x7f4dc413e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc40ae3e0 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4dc40ae7c0 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dc40af640 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40b0630 con 0x7f4dc40af640 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40b0910 con 0x7f4dc40ae7c0 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4dc40ae3e0 2023-08-22T23:21:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dca6b5640 10 monclient(hunting): authenticate will time out at 1438.954842s 2023-08-22T23:21:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dc8c52640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4dc40af640 legacy=0x7f4dc414ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dc3fff640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 msgr2=0x7f4dc413e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dc37fe640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4dc40ae7c0 legacy=0x7f4dc4141610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.705+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 0x7f4dc413e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.906+0000 7f4dc8c52640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4dc40af640 legacy=0x7f4dc414ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.906+0000 7f4dc3fff640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 msgr2=0x7f4dc413e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.906+0000 7f4dc37fe640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4dc40ae7c0 legacy=0x7f4dc4141610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:14.906+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 0x7f4dc413e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:15.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:15.307+0000 7f4dc8c52640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4dc40af640 legacy=0x7f4dc414ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:15.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:15.307+0000 7f4dc3fff640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 msgr2=0x7f4dc413e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:15.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:15.307+0000 7f4dc37fe640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4dc40ae7c0 legacy=0x7f4dc4141610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:15.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:15.307+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 0x7f4dc413e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:16.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:16.108+0000 7f4dc8c52640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4dc40af640 legacy=0x7f4dc414ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:16.108+0000 7f4dc3fff640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 msgr2=0x7f4dc413e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:16.108+0000 7f4dc37fe640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4dc40ae7c0 legacy=0x7f4dc4141610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:16.108+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 0x7f4dc413e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): tick 2023-08-22T23:21:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): _check_auth_tickets 2023-08-22T23:21:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 monclient(hunting): continuing hunt 2023-08-22T23:21:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 msgr2=0x7f4dc413e3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4dc40ae3e0 0x7f4dc413e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4dc40ae7c0 legacy=0x7f4dc4141610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4dc40af640 legacy=0x7f4dc414ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient: _add_conns ranks=[3,2] 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 0x7f4db0005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4db00036a0 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 0x7f4db00085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f4db0006110 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4db00036a0 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0008cb0 con 0x7f4db0006110 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc1ffb640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 msgr2=0x7f4db0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc37fe640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 msgr2=0x7f4db00085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 0x7f4db0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.705+0000 7f4dc37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 0x7f4db00085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:17.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.906+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 msgr2=0x7f4db0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:17.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.906+0000 7f4dc37fe640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 msgr2=0x7f4db00085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:17.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.906+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 0x7f4db0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:17.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:17.906+0000 7f4dc37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 0x7f4db00085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:18.307+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 msgr2=0x7f4db0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:18.307+0000 7f4dc37fe640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 msgr2=0x7f4db00085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:18.307+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 0x7f4db0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:18.307+0000 7f4dc37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 0x7f4db00085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:19.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:19.108+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 msgr2=0x7f4db0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:19.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:19.108+0000 7f4dc37fe640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 msgr2=0x7f4db00085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:19.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:19.108+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 0x7f4db0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:19.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:19.108+0000 7f4dc37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 0x7f4db00085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): tick 2023-08-22T23:21:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): _check_auth_tickets 2023-08-22T23:21:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 monclient(hunting): continuing hunt 2023-08-22T23:21:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-08-22T23:21:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 msgr2=0x7f4db00085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db0006110 0x7f4db00085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 msgr2=0x7f4db0005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db00036a0 0x7f4db0005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc8c52640 1 -- reap_dead start 2023-08-22T23:21:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-08-22T23:21:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4db0003b10 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4db0007ce0 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f4db000c030 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40ad980 con 0x7f4db0007ce0 2023-08-22T23:21:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40b0630 con 0x7f4db0003b10 2023-08-22T23:21:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:20.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4db000c030 2023-08-22T23:21:20.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc1ffb640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:20.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc37fe640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:20.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc8c52640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:20.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc8c52640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:20.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.705+0000 7f4dc3fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4db0003b10 legacy=0x7f4db00065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.906+0000 7f4dc37fe640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.906+0000 7f4dc8c52640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.906+0000 7f4dc8c52640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:20.906+0000 7f4dc3fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4db0003b10 legacy=0x7f4db00065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:21.307+0000 7f4dc37fe640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:21.307+0000 7f4dc8c52640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:21.307+0000 7f4dc8c52640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:21.307+0000 7f4dc3fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4db0003b10 legacy=0x7f4db00065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:22.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:22.108+0000 7f4dc37fe640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.29:6789/0 2023-08-22T23:21:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:22.108+0000 7f4dc3fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4db0003b10 legacy=0x7f4db00065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2023-08-22T23:21:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:22.108+0000 7f4dc8c52640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3301/0 2023-08-22T23:21:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:22.108+0000 7f4dc8c52640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:22.780 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:00:08.077 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:21:22.857 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:00:08.140 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:21:22.861 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:00:08.144 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:21:23.114 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:00:08.411 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:21:23.191 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:00:08.475 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:21:23.194 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:00:08.478 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:21:23.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): tick 2023-08-22T23:21:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): _check_auth_tickets 2023-08-22T23:21:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 monclient(hunting): continuing hunt 2023-08-22T23:21:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-08-22T23:21:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 -- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f4db0003b10 legacy=0x7f4db00065c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 -- >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient: _add_conns ranks=[3,4] 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4db000fa30 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f4db00104e0 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4db000fa30 2023-08-22T23:21:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db0008cb0 con 0x7f4db00104e0 2023-08-22T23:21:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc1ffb640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.705+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 msgr2=0x7f4db000fe10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.706+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.706+0000 7f4dc37fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.706+0000 7f4dc37fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:23.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.907+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 msgr2=0x7f4db000fe10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:23.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.907+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:23.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.907+0000 7f4dc37fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:23.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:23.907+0000 7f4dc37fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:24.308+0000 7f4dc3fff640 1 -- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 msgr2=0x7f4db000fe10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.29:3300/0 2023-08-22T23:21:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:24.308+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:24.308+0000 7f4dc37fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2023-08-22T23:21:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:24.308+0000 7f4dc37fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:25.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.339+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:25.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.339+0000 7f4dc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56934/0 (socket says 172.21.15.29:56934) 2023-08-22T23:21:25.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.340+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 learned_addr learned my addr 172.21.15.29:0/638926760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:25.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.340+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db000fa30 auth_method 0 2023-08-22T23:21:25.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.340+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:25.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.340+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:25.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.340+0000 7f4dc3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:25.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.388+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:25.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.416+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db00104e0 auth_method 0 2023-08-22T23:21:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.416+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.416+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.416+0000 7f4dc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.428+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 msgr2=0x7f4db000fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.428+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 msgr2=0x7f4db000fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.428+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.428+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:25.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.429+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.429+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.429+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:25.429+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): tick 2023-08-22T23:21:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): _check_auth_tickets 2023-08-22T23:21:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 monclient(hunting): continuing hunt 2023-08-22T23:21:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-08-22T23:21:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00104e0 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 msgr2=0x7f4db000fe10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db000fa30 0x7f4db000fe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc8c52640 1 -- 172.21.15.29:0/638926760 reap_dead start 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4db0007ce0 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f4db000c030 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4db000fe10 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40b0910 con 0x7f4db0007ce0 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40ad980 con 0x7f4db000fe10 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4db000c030 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc1ffb640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db000c030 auth_method 0 2023-08-22T23:21:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.706+0000 7f4dc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.707+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.707+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.707+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.707+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.908+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.908+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db000c030 auth_method 0 2023-08-22T23:21:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.908+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.908+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.908+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.918+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.918+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.918+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:26.918+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.320+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.320+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db000c030 auth_method 0 2023-08-22T23:21:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.320+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.320+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.320+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.321+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:27.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.321+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:27.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.321+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:27.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:27.321+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:28.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.121+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:28.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.121+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db000c030 auth_method 0 2023-08-22T23:21:28.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.121+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:28.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.121+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:28.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.121+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:28.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.122+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:28.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.122+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:28.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.122+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:28.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:28.122+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): tick 2023-08-22T23:21:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): _check_auth_tickets 2023-08-22T23:21:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 monclient(hunting): continuing hunt 2023-08-22T23:21:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-08-22T23:21:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 msgr2=0x7f4db000e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000c030 0x7f4db000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> v1:172.21.15.52:6789/0 conn(0x7f4db000fe10 legacy=0x7f4db000ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient: _add_conns ranks=[3,4] 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4db0005b20 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f4db00075d0 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4db0005b20 2023-08-22T23:21:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db0008cb0 con 0x7f4db00075d0 2023-08-22T23:21:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.705+0000 7f4dc1ffb640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.707+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.707+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.707+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db0005b20 auth_method 0 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.707+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.707+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.707+0000 7f4dc3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db00075d0 auth_method 0 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.708+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.709+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.709+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.709+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.709+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:29.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.909+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:29.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.910+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db0005b20 auth_method 0 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.910+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.910+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.910+0000 7f4dc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.910+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db00075d0 auth_method 0 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:29.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:29.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:29.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:29.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:29.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:29.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:29.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:29.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.911+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:29.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:29.912+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.312+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.312+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db0005b20 auth_method 0 2023-08-22T23:21:30.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.312+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.312+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.312+0000 7f4dc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:30.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.313+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:30.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db00075d0 auth_method 0 2023-08-22T23:21:30.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:30.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:30.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:30.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:30.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:30.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:30.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.314+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:30.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.315+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:30.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.315+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:30.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.315+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:30.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:30.315+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-22T23:21:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.115+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.116+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db0005b20 auth_method 0 2023-08-22T23:21:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.116+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.116+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.116+0000 7f4dc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.116+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4db00075d0 auth_method 0 2023-08-22T23:21:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.117+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.118+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-22T23:21:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.118+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.118+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:31.118+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-22T23:21:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): tick 2023-08-22T23:21:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): _check_auth_tickets 2023-08-22T23:21:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 monclient(hunting): continuing hunt 2023-08-22T23:21:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 msgr2=0x7f4db0014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db00075d0 0x7f4db0014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 msgr2=0x7f4db000e960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.29:3300/0 conn(0x7f4db0005b20 0x7f4db000e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc8c52640 1 -- 172.21.15.29:0/638926760 reap_dead start 2023-08-22T23:21:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f4db0015080 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4db000fe10 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4db0007ce0 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40b0630 con 0x7f4db0007ce0 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0005ac0 con 0x7f4db000fe10 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 1 -- 172.21.15.29:0/638926760 --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4db0015080 2023-08-22T23:21:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.705+0000 7f4dc1ffb640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.798+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.801+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db0015080 auth_method 0 2023-08-22T23:21:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.801+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.801+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.801+0000 7f4dc3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.802+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 msgr2=0x7f4db0017470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.802+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 msgr2=0x7f4db0017470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.802+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:32.802+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:33.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.109+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.110+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4db0015080 auth_method 0 2023-08-22T23:21:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.110+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.110+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.110+0000 7f4dc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:33.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.124+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 msgr2=0x7f4db0017470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-08-22T23:21:33.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.124+0000 7f4dc3fff640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 msgr2=0x7f4db0017470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-22T23:21:33.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.124+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:33.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.124+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-22T23:21:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.344+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757010770 0 0) 0x7f4dc40b0630 con 0x7f4db0007ce0 2023-08-22T23:21:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.344+0000 7f4dc2ffd640 10 monclient(hunting): _negotiate 2023-08-22T23:21:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.344+0000 7f4dc2ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.344+0000 7f4dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.344+0000 7f4dc2ffd640 10 monclient(hunting): my global_id is 4101 2023-08-22T23:21:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.344+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac003520 con 0x7f4db0007ce0 2023-08-22T23:21:33.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.392+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761003833 0 0) 0x7f4db0005ac0 con 0x7f4db000fe10 2023-08-22T23:21:33.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.392+0000 7f4dc2ffd640 10 monclient(hunting): _negotiate 2023-08-22T23:21:33.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.392+0000 7f4dc2ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.392+0000 7f4dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:33.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.392+0000 7f4dc2ffd640 10 monclient(hunting): my global_id is 4102 2023-08-22T23:21:33.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.392+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc40b0630 con 0x7f4db000fe10 2023-08-22T23:21:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3405049149 0 0) 0x7f4dac003520 con 0x7f4db0007ce0 2023-08-22T23:21:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 msgr2=0x7f4db0017470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-22T23:21:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 1 --2- 172.21.15.29:0/638926760 >> v2:172.21.15.52:3300/0 conn(0x7f4db0015080 0x7f4db0017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 >> v1:172.21.15.52:6789/0 conn(0x7f4db000fe10 legacy=0x7f4db000ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:21:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 1 monclient: found mon.noname-a-legacy 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.29:6789/0 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0005c50 con 0x7f4db0007ce0 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 10 monclient: _finish_auth 0 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.443+0000 7f4dc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dca6b5640 5 monclient: authenticate success, global_id 4101 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4066382038 0 0) 0x7f4dbc0043a0 con 0x7f4db0007ce0 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient: dump: 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:33.449 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] but i'm connected to v1:172.21.15.29:6789/0, reconnecting 2023-08-22T23:21:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 >> v1:172.21.15.29:6789/0 conn(0x7f4db0007ce0 legacy=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:21:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac003b00 0x7f4dac003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): picked mon.a con 0x7f4dac003b00 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dac0047c0 0x7f4dac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): picked mon.c con 0x7f4dac0047c0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dac0051d0 0x7f4dac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): picked mon.b con 0x7f4dac0051d0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40b0f60 con 0x7f4dac003b00 2023-08-22T23:21:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac00c100 con 0x7f4dac0051d0 2023-08-22T23:21:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac00c2c0 con 0x7f4dac0047c0 2023-08-22T23:21:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dc2ffd640 10 monclient(hunting): _finish_auth 0 2023-08-22T23:21:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.444+0000 7f4dca6b5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.478+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac003b00 0x7f4dac003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.478+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dac0047c0 0x7f4dac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.478+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4dac003b00 auth_method 0 2023-08-22T23:21:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.478+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.478+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.478+0000 7f4dc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:33.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-08-22T23:21:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4dac0047c0 auth_method 0 2023-08-22T23:21:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.484+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.485+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.485+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.486+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-08-22T23:21:33.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_done global_id 4101 payload 293 2023-08-22T23:21:33.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dac0051d0 msgr2=0x7f4dac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:33.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dac0051d0 0x7f4dac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac003b00 msgr2=0x7f4dac003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:33.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac003b00 0x7f4dac003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 10 monclient: _finish_hunting 0 2023-08-22T23:21:33.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 1 monclient: found mon.c 2023-08-22T23:21:33.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:33.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:33.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 1 -- 172.21.15.29:0/638926760 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac003520 con 0x7f4dac0047c0 2023-08-22T23:21:33.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 10 monclient: _finish_auth 0 2023-08-22T23:21:33.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 10 monclient: _check_auth_tickets 2023-08-22T23:21:33.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:33.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.488+0000 7f4dc8c52640 1 -- 172.21.15.29:0/638926760 reap_dead start 2023-08-22T23:21:33.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.503+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dac0047c0 0x7f4dac004c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4db000f8f0 tx=0x7f4dac0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce3f341775f07957 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8003c30 con 0x7f4dac0047c0 2023-08-22T23:21:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: dump: 2023-08-22T23:21:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dca6b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db800c040 con 0x7f4dac0047c0 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dca6b5640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dca6b5640 4 set_mon_vals no callback set 2023-08-22T23:21:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 1 -- 172.21.15.29:0/638926760 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db802d970 con 0x7f4dac0047c0 2023-08-22T23:21:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.573+0000 7f4dc2ffd640 10 monclient: dump: 2023-08-22T23:21:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:33.579 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 10 monclient: shutdown 2023-08-22T23:21:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 -- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dac0047c0 msgr2=0x7f4dac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dac0047c0 0x7f4dac004c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4db000f8f0 tx=0x7f4dac0057f0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 -- 172.21.15.29:0/638926760 shutdown_connections 2023-08-22T23:21:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 --2- 172.21.15.29:0/638926760 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dac0047c0 0x7f4dac004c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 -- 172.21.15.29:0/638926760 >> 172.21.15.29:0/638926760 conn(0x7f4dc40b69a0 msgr2=0x7f4dc40b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 -- 172.21.15.29:0/638926760 shutdown_connections 2023-08-22T23:21:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 1 -- 172.21.15.29:0/638926760 wait complete. 2023-08-22T23:21:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 10 monclient: build_initial_monmap 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.574+0000 7f4dca6b5640 10 monclient: monmap: 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 1 librados: starting msgr at 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 1 librados: starting objecter 2023-08-22T23:21:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 1 Processor -- start 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 1 -- start start 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 1 librados: setting wanted keys 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 1 librados: calling monclient init 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.575+0000 7f4dca6b5640 10 monclient: init 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4dc4139b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): picked mon.noname-a con 0x7f4dac0047c0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc40a74e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4dc41d95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc4139010 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4141e80 con 0x7f4dac0047c0 2023-08-22T23:21:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40a7940 con 0x7f4dc4139010 2023-08-22T23:21:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc41422a0 con 0x7f4dc40a74e0 2023-08-22T23:21:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.576+0000 7f4dca6b5640 10 monclient(hunting): authenticate will time out at 1457.826605s 2023-08-22T23:21:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.578+0000 7f4dc8c52640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4dc41d95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.584+0000 7f4dc37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.585+0000 7f4dc37fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:37256/0 (socket says 172.21.15.29:37256) 2023-08-22T23:21:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.585+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 learned_addr learned my addr 172.21.15.29:0/1654322969 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.585+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4dc40a74e0 auth_method 0 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.585+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.585+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.585+0000 7f4dc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.588+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.588+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.588+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:33.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2023-08-22T23:21:33.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 msgr2=0x7f4dc41d95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:33.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4dc41d95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 msgr2=0x7f4dc4139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4dc4139b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 10 monclient: _finish_hunting 0 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 1 monclient: found mon.noname-c 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.600+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40a8770 con 0x7f4dc40a74e0 2023-08-22T23:21:33.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.602+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4db80039b0 tx=0x7f4db8005a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.602+0000 7f4dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f4dc40a74e0 2023-08-22T23:21:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.602+0000 7f4dc0ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db80052d0 con 0x7f4dc40a74e0 2023-08-22T23:21:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: dump: 2023-08-22T23:21:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: _finish_auth 0 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db8005450 con 0x7f4dc40a74e0 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8005800 con 0x7f4dc40a74e0 2023-08-22T23:21:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 5 monclient: authenticate success, global_id 4100 2023-08-22T23:21:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc0ff9640 10 monclient: dump: 2023-08-22T23:21:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 10 monclient: _renew_subs 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dc27fc640 4 set_mon_vals no callback set 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc40a89e0 con 0x7f4dc40a74e0 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 10 monclient: _renew_subs 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc4139490 con 0x7f4dc40a74e0 2023-08-22T23:21:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.607+0000 7f4dca6b5640 1 librados: init done 2023-08-22T23:21:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.608+0000 7f4dca6b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.608+0000 7f4dca6b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.608+0000 7f4dca6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.608+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc404dbd0 con 0x7f4dc40a74e0 2023-08-22T23:21:33.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 msgr2=0x7f4dc4138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-08-22T23:21:33.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 msgr2=0x7f4dc4138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4db80039b0 tx=0x7f4db8005a50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4db80039b0 tx=0x7f4db8005a50 comp rx=0 tx=0).stop 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 msgr2=0x7f4dc4138ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4dc40a74e0 0x7f4dc4138ad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4db00080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): picked mon.b con 0x7f4dc4139010 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4db0006cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): picked mon.a con 0x7f4dac0047c0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000fe10 0x7f4db000ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): picked mon.c con 0x7f4db000fe10 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0007f00 con 0x7f4dac0047c0 2023-08-22T23:21:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db000f200 con 0x7f4dc4139010 2023-08-22T23:21:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db000f370 con 0x7f4db000fe10 2023-08-22T23:21:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.626+0000 7f4dc0ff9640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:33.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.631+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4db0006cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.638+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4db00080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:33.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.638+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request con 0x7f4dc4139010 auth_method 0 2023-08-22T23:21:33.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.638+0000 7f4dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:33.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.638+0000 7f4dc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.638+0000 7f4dc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:33.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.640+0000 7f4dc3fff640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 msgr2=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-08-22T23:21:33.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.640+0000 7f4dc3fff640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 msgr2=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-22T23:21:33.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.640+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4db00080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-22T23:21:33.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.640+0000 7f4dc3fff640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4db00080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-22T23:21:33.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request con 0x7f4dac0047c0 auth_method 0 2023-08-22T23:21:33.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:33.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:33.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:21:33.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:33.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:33.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.646+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-08-22T23:21:33.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient(hunting): handle_auth_done global_id 4100 payload 293 2023-08-22T23:21:33.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000fe10 msgr2=0x7f4db000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:33.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000fe10 0x7f4db000ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 msgr2=0x7f4db00080e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-22T23:21:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4db00080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient: _finish_hunting 0 2023-08-22T23:21:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 monclient: found mon.a 2023-08-22T23:21:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4db000f5a0 con 0x7f4dac0047c0 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc4139490 con 0x7f4dac0047c0 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient: _finish_auth 0 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 10 monclient: _check_auth_tickets 2023-08-22T23:21:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.648+0000 7f4dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.654+0000 7f4dc37fe640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4db0006cb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4db8006d10 tx=0x7f4db8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.654+0000 7f4dc0ff9640 10 client.4100.objecter ms_handle_connect 0x7f4dac0047c0 2023-08-22T23:21:33.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.654+0000 7f4dc0ff9640 10 client.4100.objecter resend_mon_ops 2023-08-22T23:21:33.751 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:33.750+0000 7f4deef5b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-22T23:21:33.769 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:33.767+0000 7f27b0a3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-22T23:21:33.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8051440 con 0x7f4dac0047c0 2023-08-22T23:21:33.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:33.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:21:33.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 monclient: dump: 2023-08-22T23:21:33.850 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:33.850 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:33.850 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.850 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:33.850 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db80515a0 con 0x7f4dac0047c0 2023-08-22T23:21:33.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4db8051710 con 0x7f4dac0047c0 2023-08-22T23:21:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4db803e8a0 con 0x7f4dac0047c0 2023-08-22T23:21:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 client.4100.objecter ms_dispatch 0x7f4dc4000b90 osd_map(1..1 src has 1..1) v4 2023-08-22T23:21:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2023-08-22T23:21:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 3 client.4100.objecter handle_osd_map decoding full epoch 1 2023-08-22T23:21:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc1ffb640 4 set_mon_vals no callback set 2023-08-22T23:21:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 20 client.4100.objecter dump_active .. 0 homeless 2023-08-22T23:21:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f4db8055040 con 0x7f4dac0047c0 2023-08-22T23:21:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.846+0000 7f4dc0ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:33.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.912+0000 7f4dca6b5640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-08-22T23:21:33.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.912+0000 7f4dca6b5640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-08-22T23:21:33.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.912+0000 7f4dca6b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:33.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:33.912+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4dc404dbd0 con 0x7f4dac0047c0 2023-08-22T23:21:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.031+0000 7f4dc0ff9640 1 -- 172.21.15.29:0/1654322969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4db8042020 con 0x7f4dac0047c0 2023-08-22T23:21:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.031+0000 7f4dc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-08-22T23:21:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.031+0000 7f4dc0ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-08-22T23:21:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-08-22T23:21:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 10 librados: watch_flush enter 2023-08-22T23:21:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 10 librados: watch_flush exit 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 20 client.4100.objecter shutdown clearing up homeless session... 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 10 client.4100.objecter successfully canceled tick 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 10 monclient: shutdown 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 msgr2=0x7f4db0006cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4db0006cb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4db8006d10 tx=0x7f4db8031000 comp rx=0 tx=0).stop 2023-08-22T23:21:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 shutdown_connections 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4db000fe10 0x7f4db000ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dac0047c0 0x7f4db0006cb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 --2- 172.21.15.29:0/1654322969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4dc4139010 0x7f4db00080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 >> 172.21.15.29:0/1654322969 conn(0x7f4dc40b69a0 msgr2=0x7f4dc40b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 shutdown_connections 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.032+0000 7f4dca6b5640 1 -- 172.21.15.29:0/1654322969 wait complete. 2023-08-22T23:21:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:34.033+0000 7f4dca6b5640 1 librados: shutdown 2023-08-22T23:21:34.045 INFO:tasks.ceph:check_enable_crimson: False 2023-08-22T23:21:34.045 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-08-22T23:21:34.046 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:34.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-08-22T23:21:34.077 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:34.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-08-22T23:21:34.147 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:34.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-08-22T23:21:34.221 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:21:34.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-08-22T23:21:34.293 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:34.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-08-22T23:21:34.324 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:34.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-08-22T23:21:34.395 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:34.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-08-22T23:21:34.466 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2023-08-22T23:21:34.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-08-22T23:21:34.536 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new d051e519-9506-4291-a28a-9400b04f3884 0 2023-08-22T23:21:34.611 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:34.610+0000 7f4deef5b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-22T23:21:34.696 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.695+0000 7f4994845640 10 monclient: get_monmap_and_config 2023-08-22T23:21:34.696 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.695+0000 7f4994845640 10 monclient: build_initial_monmap 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.695+0000 7f4994845640 10 monclient: monmap: 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:34.696409+0000 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:34.696409+0000 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:34.697 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:34.698 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:34.698 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:34.698 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:34.698 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:34.698 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:34.700 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 Processor -- start 2023-08-22T23:21:34.700 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 -- start start 2023-08-22T23:21:34.700 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient: init 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49901445a0 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49900aff50 0x7f49900b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): picked mon.noname-a con 0x7f49900aff50 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f49900a7c50 0x7f49900a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:34.701 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): picked mon.noname-c con 0x7f49900a7c50 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49900a97c0 con 0x7f49901445a0 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49900a9aa0 con 0x7f49900aff50 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4990145130 con 0x7f49900a7c50 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:34.702 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.696+0000 7f4994845640 10 monclient(hunting): authenticate will time out at 1483.909677s 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f4987fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49900aff50 0x7f49900b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498ed9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f49900a7c50 0x7f49900a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f49901445a0 0x7f4990144980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.52:51468/0 (socket says 172.21.15.52:51468) 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498e59c640 1 -- 172.21.15.52:0/2870926568 learned_addr learned my addr 172.21.15.52:0/2870926568 (peer_addr_for_me v1:172.21.15.52:0/0) 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498ed9d640 10 monclient(hunting): get_auth_request con 0x7f49900a7c50 auth_method 0 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498ed9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:34.703 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.697+0000 7f498ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:34.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f4987fff640 10 monclient(hunting): get_auth_request con 0x7f49900aff50 auth_method 0 2023-08-22T23:21:34.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f4987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:34.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f4987fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:34.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f4987fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:34.704 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f498ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:34.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f498ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:34.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.698+0000 7f498ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:34.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.699+0000 7f4987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:34.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.699+0000 7f4987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:34.705 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.699+0000 7f4987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:34.706 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.699+0000 7f498ed9d640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2023-08-22T23:21:34.706 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 1 -- 172.21.15.52:0/2870926568 >> v2:172.21.15.29:3300/0 conn(0x7f49900aff50 msgr2=0x7f49900b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 1 --2- 172.21.15.52:0/2870926568 >> v2:172.21.15.29:3300/0 conn(0x7f49900aff50 0x7f49900b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 1 -- 172.21.15.52:0/2870926568 >> v1:172.21.15.29:6789/0 conn(0x7f49901445a0 legacy=0x7f4990144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 10 monclient: _finish_hunting 0 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 1 monclient: found mon.noname-c 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:21:34.707 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.700+0000 7f498ed9d640 1 -- 172.21.15.52:0/2870926568 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49900b0b40 con 0x7f49900a7c50 2023-08-22T23:21:34.708 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.702+0000 7f498ed9d640 1 --2- 172.21.15.52:0/2870926568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49900a7c50 0x7f49900a8090 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f49900a9f60 tx=0x7f498802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467a3aa5eb5934e0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:34.709 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 1 -- 172.21.15.52:0/2870926568 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498802cc80 con 0x7f49900a7c50 2023-08-22T23:21:34.709 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: dump: 2023-08-22T23:21:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:34.711 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: _finish_auth 0 2023-08-22T23:21:34.712 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: _check_auth_tickets 2023-08-22T23:21:34.712 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:34.712 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 1 -- 172.21.15.52:0/2870926568 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498802cde0 con 0x7f49900a7c50 2023-08-22T23:21:34.712 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:34.712 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f4994845640 5 monclient: authenticate success, global_id 4111 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 1 -- 172.21.15.52:0/2870926568 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988034660 con 0x7f49900a7c50 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f4994845640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f4994845640 4 set_mon_vals no callback set 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.706+0000 7f498dd9b640 10 monclient: dump: 2023-08-22T23:21:34.713 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:34.714 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:34.714 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.714 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.714 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:34.714 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:34.714 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:34.715 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:34.715 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:34.715 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:34.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 10 monclient: shutdown 2023-08-22T23:21:34.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:34.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 -- 172.21.15.52:0/2870926568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49900a7c50 msgr2=0x7f49900a8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:34.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 --2- 172.21.15.52:0/2870926568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49900a7c50 0x7f49900a8090 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f49900a9f60 tx=0x7f498802bed0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 -- 172.21.15.52:0/2870926568 shutdown_connections 2023-08-22T23:21:34.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 --2- 172.21.15.52:0/2870926568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49900a7c50 0x7f49900a8090 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 --2- 172.21.15.52:0/2870926568 >> v2:172.21.15.29:3300/0 conn(0x7f49900aff50 0x7f49900b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 -- 172.21.15.52:0/2870926568 >> 172.21.15.52:0/2870926568 conn(0x7f49900b6910 msgr2=0x7f49900b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:34.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 -- 172.21.15.52:0/2870926568 shutdown_connections 2023-08-22T23:21:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.707+0000 7f4994845640 1 -- 172.21.15.52:0/2870926568 wait complete. 2023-08-22T23:21:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 10 monclient: build_initial_monmap 2023-08-22T23:21:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 10 monclient: monmap: 2023-08-22T23:21:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:34.718 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 1 librados: starting msgr at 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 1 librados: starting objecter 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 1 Processor -- start 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 1 -- start start 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 1 librados: setting wanted keys 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 1 librados: calling monclient init 2023-08-22T23:21:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.708+0000 7f4994845640 10 monclient: init 2023-08-22T23:21:34.720 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:34.707+0000 7f27b0a3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-22T23:21:34.722 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:34.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:21:34.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49900a7c50 0x7f4990140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:34.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): picked mon.noname-a con 0x7f49900a7c50 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:34.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f49900aff50 0x7f4990140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:34.723 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): picked mon.noname-c con 0x7f49900aff50 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 0x7f49901d5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): picked mon.noname-b con 0x7f49901445a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901419b0 con 0x7f49900a7c50 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49900ac740 con 0x7f49901445a0 2023-08-22T23:21:34.724 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49900ac900 con 0x7f49900aff50 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4994845640 10 monclient(hunting): authenticate will time out at 1483.922116s 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 0x7f49901d5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f4987fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f49900aff50 0x7f4990140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.709+0000 7f498e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49900a7c50 0x7f4990140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f49900aff50 0x7f4990140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37246/0 (socket says 172.21.15.52:37246) 2023-08-22T23:21:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 0x7f49901d5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35836/0 (socket says 172.21.15.52:35836) 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 1 -- 172.21.15.52:0/215311928 learned_addr learned my addr 172.21.15.52:0/215311928 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): get_auth_request con 0x7f49900aff50 auth_method 0 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): get_auth_request con 0x7f49901445a0 auth_method 0 2023-08-22T23:21:34.726 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498e59c640 10 monclient(hunting): get_auth_request con 0x7f49900a7c50 auth_method 0 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498e59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498e59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f498ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.710+0000 7f4987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.711+0000 7f498e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.711+0000 7f498e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.711+0000 7f498e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 1 -- 172.21.15.52:0/215311928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f49900aff50 msgr2=0x7f4990140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f49900aff50 0x7f4990140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 1 -- 172.21.15.52:0/215311928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49900a7c50 msgr2=0x7f4990140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49900a7c50 0x7f4990140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 10 monclient: _finish_hunting 0 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 1 monclient: found mon.noname-b 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:34.729 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.713+0000 7f498ed9d640 1 -- 172.21.15.52:0/215311928 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988009a70 con 0x7f49901445a0 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.715+0000 7f498ed9d640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 0x7f49901d5e90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4988033000 tx=0x7f4988004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.715+0000 7f4986ffd640 10 client.?.objecter ms_handle_connect 0x7f49901445a0 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.715+0000 7f4986ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498803e610 con 0x7f49901445a0 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:34.730 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: dump: 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:34.731 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:34.732 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:34.732 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:34.732 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:34.732 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:34.732 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: _finish_auth 0 2023-08-22T23:21:34.732 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:34.733 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:34.733 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498803e770 con 0x7f49901445a0 2023-08-22T23:21:34.733 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:34.733 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498803eb20 con 0x7f49901445a0 2023-08-22T23:21:34.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 5 monclient: authenticate success, global_id 4114 2023-08-22T23:21:34.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:34.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:34.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4986ffd640 10 monclient: dump: 2023-08-22T23:21:34.734 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:34.734 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f498d59a640 4 set_mon_vals no callback set 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 10 monclient: _renew_subs 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49901414b0 con 0x7f49901445a0 2023-08-22T23:21:34.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:34.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 10 monclient: _renew_subs 2023-08-22T23:21:34.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:34.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49901d65b0 con 0x7f49901445a0 2023-08-22T23:21:34.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.719+0000 7f4994845640 1 librados: init done 2023-08-22T23:21:34.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.720+0000 7f4994845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:34.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.720+0000 7f4994845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:34.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.720+0000 7f4994845640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:34.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.720+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4990141af0 con 0x7f49901445a0 2023-08-22T23:21:34.772 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:34.771+0000 7f4deef5b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-22T23:21:34.824 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:34.823+0000 7f4deef5b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-22T23:21:34.858 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:34.856+0000 7f27b0a3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-22T23:21:34.871 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:34.870+0000 7f4deef5b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-22T23:21:34.903 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f498803b050 con 0x7f49901445a0 2023-08-22T23:21:34.903 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4988049450 con 0x7f49901445a0 2023-08-22T23:21:34.903 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 10 client.4114.objecter ms_dispatch 0x7f4990000b90 osd_map(2..2 src has 1..2) v4 2023-08-22T23:21:34.904 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2023-08-22T23:21:34.904 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2023-08-22T23:21:34.904 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 20 client.4114.objecter dump_active .. 0 homeless 2023-08-22T23:21:34.904 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f4990141af0 con 0x7f49901445a0 2023-08-22T23:21:34.904 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:34.904 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.902+0000 7f4986ffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:34.909 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:34.908+0000 7f27b0a3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-22T23:21:34.955 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:34.954+0000 7f27b0a3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-22T23:21:34.977 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.976+0000 7f4994845640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d051e519-9506-4291-a28a-9400b04f3884", "id": 0}] 2023-08-22T23:21:34.977 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.976+0000 7f4994845640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d051e519-9506-4291-a28a-9400b04f3884", "id": 0}] 2023-08-22T23:21:34.977 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.976+0000 7f4994845640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:34.978 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:34.976+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d051e519-9506-4291-a28a-9400b04f3884", "id": 0} v 0) v1 -- 0x7f49900b7220 con 0x7f49901445a0 2023-08-22T23:21:35.187 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.186+0000 7f4986ffd640 1 -- 172.21.15.52:0/215311928 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d051e519-9506-4291-a28a-9400b04f3884", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f498804e060 con 0x7f49901445a0 2023-08-22T23:21:35.187 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.186+0000 7f4986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d051e519-9506-4291-a28a-9400b04f3884", "id": 0}] 2023-08-22T23:21:35.187 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.186+0000 7f4986ffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:35.188 INFO:teuthology.orchestra.run.smithi052.stdout:0 2023-08-22T23:21:35.189 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.186+0000 7f4994845640 10 librados: watch_flush enter 2023-08-22T23:21:35.189 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 10 librados: watch_flush exit 2023-08-22T23:21:35.189 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 20 client.4114.objecter shutdown clearing up homeless session... 2023-08-22T23:21:35.189 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 10 client.4114.objecter successfully canceled tick 2023-08-22T23:21:35.189 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 10 monclient: shutdown 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 msgr2=0x7f49901d5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 0x7f49901d5e90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4988033000 tx=0x7f4988004a40 comp rx=0 tx=0).stop 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 shutdown_connections 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49901445a0 0x7f49901d5e90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f49900aff50 0x7f4990140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 --2- 172.21.15.52:0/215311928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49900a7c50 0x7f4990140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.190 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 >> 172.21.15.52:0/215311928 conn(0x7f49900b6910 msgr2=0x7f49900b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:35.191 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 shutdown_connections 2023-08-22T23:21:35.191 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 -- 172.21.15.52:0/215311928 wait complete. 2023-08-22T23:21:35.191 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.187+0000 7f4994845640 1 librados: shutdown 2023-08-22T23:21:35.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 26caaa51-e454-4283-a40b-44550023da15 1 2023-08-22T23:21:35.342 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.341+0000 7efe87fa4640 10 monclient: get_monmap_and_config 2023-08-22T23:21:35.342 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.341+0000 7efe87fa4640 10 monclient: build_initial_monmap 2023-08-22T23:21:35.342 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.341+0000 7efe87fa4640 10 monclient: monmap: 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:35.342058+0000 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:35.342058+0000 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.343 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:35.344 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:35.344 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:35.344 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:35.344 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:35.344 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.341+0000 7efe87fa4640 1 Processor -- start 2023-08-22T23:21:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.341+0000 7efe87fa4640 1 -- start start 2023-08-22T23:21:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.341+0000 7efe87fa4640 10 monclient: init 2023-08-22T23:21:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-08-22T23:21:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): picked mon.noname-b con 0x7efe801448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe800add90 0x7efe80145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): picked mon.noname-a con 0x7efe800add90 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe8013e190 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe800b12d0 con 0x7efe8013e190 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800afae0 con 0x7efe800add90 2023-08-22T23:21:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800afdc0 con 0x7efe801448e0 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.342+0000 7efe87fa4640 10 monclient(hunting): authenticate will time out at 1484.555339s 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe8553f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe800add90 0x7efe80145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37252/0 (socket says 172.21.15.52:37252) 2023-08-22T23:21:35.348 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 1 -- 172.21.15.52:0/1996757506 learned_addr learned my addr 172.21.15.52:0/1996757506 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 10 monclient(hunting): get_auth_request con 0x7efe801448e0 auth_method 0 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe85d40640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe8553f640 10 monclient(hunting): get_auth_request con 0x7efe800add90 auth_method 0 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe8553f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe8553f640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.343+0000 7efe8553f640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.344+0000 7efe85d40640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.344+0000 7efe85d40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.344+0000 7efe85d40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe8553f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe8553f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe8553f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2023-08-22T23:21:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 1 -- 172.21.15.52:0/1996757506 >> v2:172.21.15.29:3300/0 conn(0x7efe800add90 msgr2=0x7efe80145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 1 --2- 172.21.15.52:0/1996757506 >> v2:172.21.15.29:3300/0 conn(0x7efe800add90 0x7efe80145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 1 -- 172.21.15.52:0/1996757506 >> v1:172.21.15.29:6789/0 conn(0x7efe8013e190 legacy=0x7efe801410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 10 monclient: _finish_hunting 0 2023-08-22T23:21:35.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 1 monclient: found mon.noname-b 2023-08-22T23:21:35.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:35.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.345+0000 7efe85d40640 1 -- 172.21.15.52:0/1996757506 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800b00a0 con 0x7efe801448e0 2023-08-22T23:21:35.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.348+0000 7efe85d40640 1 --2- 172.21.15.52:0/1996757506 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80144cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe74009870 tx=0x7efe7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5112e9621d206df server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:35.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 1 -- 172.21.15.52:0/1996757506 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74030840 con 0x7efe801448e0 2023-08-22T23:21:35.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: dump: 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: _finish_auth 0 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: _check_auth_tickets 2023-08-22T23:21:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 1 -- 172.21.15.52:0/1996757506 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe740309a0 con 0x7efe801448e0 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 1 -- 172.21.15.52:0/1996757506 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74030d30 con 0x7efe801448e0 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe87fa4640 5 monclient: authenticate success, global_id 4106 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe87fa4640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe87fa4640 4 set_mon_vals no callback set 2023-08-22T23:21:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.352+0000 7efe84d3e640 10 monclient: dump: 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.357 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.358 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.358 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.358 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.358 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.361 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 10 monclient: shutdown 2023-08-22T23:21:35.361 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:35.361 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 -- 172.21.15.52:0/1996757506 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 msgr2=0x7efe80144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.361 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 --2- 172.21.15.52:0/1996757506 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80144cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe74009870 tx=0x7efe7402b980 comp rx=0 tx=0).stop 2023-08-22T23:21:35.361 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 -- 172.21.15.52:0/1996757506 shutdown_connections 2023-08-22T23:21:35.361 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 --2- 172.21.15.52:0/1996757506 >> v2:172.21.15.29:3300/0 conn(0x7efe800add90 0x7efe80145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 --2- 172.21.15.52:0/1996757506 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80144cc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 -- 172.21.15.52:0/1996757506 >> 172.21.15.52:0/1996757506 conn(0x7efe80065ed0 msgr2=0x7efe800b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 -- 172.21.15.52:0/1996757506 shutdown_connections 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 1 -- 172.21.15.52:0/1996757506 wait complete. 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 10 monclient: build_initial_monmap 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.353+0000 7efe87fa4640 10 monclient: monmap: 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:35.362 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:35.363 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 librados: starting msgr at 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 librados: starting objecter 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 Processor -- start 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 -- start start 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 librados: setting wanted keys 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 librados: calling monclient init 2023-08-22T23:21:35.364 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient: init 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 0x7efe8013f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): picked mon.noname-a con 0x7efe800add90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efe8013e190 0x7efe80135d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): picked mon.noname-b con 0x7efe8013e190 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.365 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): picked mon.noname-c con 0x7efe801448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe80143fe0 con 0x7efe800add90 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe80143ac0 con 0x7efe8013e190 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe80144420 con 0x7efe801448e0 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.354+0000 7efe87fa4640 10 monclient(hunting): authenticate will time out at 1484.567614s 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe86541640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 0x7efe8013f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe8553f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efe8013e190 0x7efe80135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 0x7efe8013f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.52:39296/0 (socket says 172.21.15.52:39296) 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 1 -- 172.21.15.52:0/858274112 learned_addr learned my addr 172.21.15.52:0/858274112 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 10 monclient(hunting): get_auth_request con 0x7efe800add90 auth_method 0 2023-08-22T23:21:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe85d40640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe86541640 10 monclient(hunting): get_auth_request con 0x7efe801448e0 auth_method 0 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe86541640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe86541640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe86541640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe8553f640 10 monclient(hunting): get_auth_request con 0x7efe8013e190 auth_method 0 2023-08-22T23:21:35.368 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe8553f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe8553f640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.355+0000 7efe8553f640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe85d40640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe85d40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe85d40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe8553f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe8553f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.369 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe8553f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe86541640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe86541640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.356+0000 7efe86541640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 10 monclient(hunting): handle_auth_done global_id 4122 payload 824 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 1 -- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 msgr2=0x7efe80133aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80133aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 1 -- 172.21.15.52:0/858274112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efe8013e190 msgr2=0x7efe80135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efe8013e190 0x7efe80135d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 10 monclient: _finish_hunting 0 2023-08-22T23:21:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 1 monclient: found mon.noname-a 2023-08-22T23:21:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:21:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.359+0000 7efe85d40640 1 -- 172.21.15.52:0/858274112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740094d0 con 0x7efe800add90 2023-08-22T23:21:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.361+0000 7efe85d40640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 0x7efe8013f160 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efe740090b0 tx=0x7efe740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.361+0000 7efe727fc640 10 client.?.objecter ms_handle_connect 0x7efe800add90 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.361+0000 7efe727fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7403e070 con 0x7efe800add90 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: dump: 2023-08-22T23:21:35.373 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: _finish_auth 0 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: _check_auth_tickets 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe74039a10 con 0x7efe800add90 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74039de0 con 0x7efe800add90 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe727fc640 10 monclient: dump: 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.376 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 5 monclient: authenticate success, global_id 4122 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 10 monclient: _renew_subs 2023-08-22T23:21:35.377 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe80143630 con 0x7efe800add90 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe73fff640 4 set_mon_vals no callback set 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 10 monclient: _renew_subs 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe801341c0 con 0x7efe800add90 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.365+0000 7efe87fa4640 1 librados: init done 2023-08-22T23:21:35.378 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.366+0000 7efe87fa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.366+0000 7efe87fa4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.366+0000 7efe87fa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.366+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80066cb0 con 0x7efe800add90 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.370+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efe74055730 con 0x7efe800add90 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.370+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7efe7404b940 con 0x7efe800add90 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.370+0000 7efe727fc640 10 client.4122.objecter ms_dispatch 0x7efe80000b90 osd_map(3..3 src has 1..3) v4 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.370+0000 7efe727fc640 3 client.4122.objecter handle_osd_map got epochs [3,3] > 0 2023-08-22T23:21:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.370+0000 7efe727fc640 3 client.4122.objecter handle_osd_map decoding full epoch 3 2023-08-22T23:21:35.380 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.370+0000 7efe727fc640 20 client.4122.objecter dump_active .. 0 homeless 2023-08-22T23:21:35.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.473+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7efe74034070 con 0x7efe800add90 2023-08-22T23:21:35.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.473+0000 7efe727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:35.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.473+0000 7efe727fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:35.547 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.546+0000 7efe87fa4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26caaa51-e454-4283-a40b-44550023da15", "id": 1}] 2023-08-22T23:21:35.548 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.546+0000 7efe87fa4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26caaa51-e454-4283-a40b-44550023da15", "id": 1}] 2023-08-22T23:21:35.548 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.546+0000 7efe87fa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:35.548 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.546+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26caaa51-e454-4283-a40b-44550023da15", "id": 1} v 0) v1 -- 0x7efe80143c80 con 0x7efe800add90 2023-08-22T23:21:35.610 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.609+0000 7efe727fc640 1 -- 172.21.15.52:0/858274112 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26caaa51-e454-4283-a40b-44550023da15", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7efe7404bc30 con 0x7efe800add90 2023-08-22T23:21:35.611 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.609+0000 7efe727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26caaa51-e454-4283-a40b-44550023da15", "id": 1}] 2023-08-22T23:21:35.611 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.609+0000 7efe727fc640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:35.611 INFO:teuthology.orchestra.run.smithi052.stdout:1 2023-08-22T23:21:35.612 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 10 librados: watch_flush enter 2023-08-22T23:21:35.612 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 10 librados: watch_flush exit 2023-08-22T23:21:35.612 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 20 client.4122.objecter shutdown clearing up homeless session... 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 10 client.4122.objecter successfully canceled tick 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 10 monclient: shutdown 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 msgr2=0x7efe8013f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 0x7efe8013f160 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efe740090b0 tx=0x7efe740046c0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 shutdown_connections 2023-08-22T23:21:35.613 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efe801448e0 0x7efe80133aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.614 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efe8013e190 0x7efe80135d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.614 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 --2- 172.21.15.52:0/858274112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe800add90 0x7efe8013f160 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.614 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 >> 172.21.15.52:0/858274112 conn(0x7efe80065ed0 msgr2=0x7efe80144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:35.614 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 shutdown_connections 2023-08-22T23:21:35.614 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 -- 172.21.15.52:0/858274112 wait complete. 2023-08-22T23:21:35.614 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.610+0000 7efe87fa4640 1 librados: shutdown 2023-08-22T23:21:35.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 0a17913e-6533-4e19-a9c7-edd21971b8e5 2 2023-08-22T23:21:35.637 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:35.635+0000 7f4deef5b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-22T23:21:35.697 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:35.694+0000 7f4deef5b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-22T23:21:35.731 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.729+0000 7f71c721b640 10 monclient: get_monmap_and_config 2023-08-22T23:21:35.731 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.729+0000 7f71c721b640 10 monclient: build_initial_monmap 2023-08-22T23:21:35.731 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.729+0000 7f71c721b640 10 monclient: monmap: 2023-08-22T23:21:35.731 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:35.731 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:35.730794+0000 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:35.730794+0000 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:35.732 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:35.733 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:35.733 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:35.733 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.734 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.730+0000 7f71c721b640 1 Processor -- start 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.730+0000 7f71c721b640 1 -- start start 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.730+0000 7f71c721b640 10 monclient: init 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71c01448e0 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71c00add90 0x7f71c013e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.735 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): picked mon.noname-a con 0x7f71c00add90 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c014ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): picked mon.noname-b con 0x7f71c013e6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c00b12d0 con 0x7f71c01448e0 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c00afae0 con 0x7f71c00add90 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.736 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c00afdc0 con 0x7f71c013e6d0 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.731+0000 7f71c721b640 10 monclient(hunting): authenticate will time out at 1484.944165s 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71bffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71c00add90 0x7f71c013e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c014ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c4fb7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f71c01448e0 0x7f71c0144cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.52:51480/0 (socket says 172.21.15.52:51480) 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c4fb7640 1 -- 172.21.15.52:0/1148764369 learned_addr learned my addr 172.21.15.52:0/1148764369 (peer_addr_for_me v1:172.21.15.52:0/0) 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): get_auth_request con 0x7f71c013e6d0 auth_method 0 2023-08-22T23:21:35.737 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71bffff640 10 monclient(hunting): get_auth_request con 0x7f71c00add90 auth_method 0 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71bffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71bffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.739 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.732+0000 7f71c57b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.739 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.733+0000 7f71bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.739 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.733+0000 7f71bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.739 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.733+0000 7f71bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.739 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-08-22T23:21:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 1 -- 172.21.15.52:0/1148764369 >> v2:172.21.15.29:3300/0 conn(0x7f71c00add90 msgr2=0x7f71c013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 1 --2- 172.21.15.52:0/1148764369 >> v2:172.21.15.29:3300/0 conn(0x7f71c00add90 0x7f71c013e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 1 -- 172.21.15.52:0/1148764369 >> v1:172.21.15.29:6789/0 conn(0x7f71c01448e0 legacy=0x7f71c0144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 10 monclient: _finish_hunting 0 2023-08-22T23:21:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 1 monclient: found mon.noname-b 2023-08-22T23:21:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.734+0000 7f71c57b8640 1 -- 172.21.15.52:0/1148764369 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c00b00a0 con 0x7f71c013e6d0 2023-08-22T23:21:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.736+0000 7f71c57b8640 1 --2- 172.21.15.52:0/1148764369 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c014ba00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009d90 tx=0x7f71b802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60666db047c1423 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 1 -- 172.21.15.52:0/1148764369 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b802cc50 con 0x7f71c013e6d0 2023-08-22T23:21:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: dump: 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: _finish_auth 0 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 1 -- 172.21.15.52:0/1148764369 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b802cdb0 con 0x7f71c013e6d0 2023-08-22T23:21:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 1 -- 172.21.15.52:0/1148764369 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8034660 con 0x7f71c013e6d0 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71bf7fe640 10 monclient: dump: 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.745 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71c721b640 5 monclient: authenticate success, global_id 4112 2023-08-22T23:21:35.746 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71c721b640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:35.747 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71c721b640 4 set_mon_vals no callback set 2023-08-22T23:21:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71c721b640 10 monclient: shutdown 2023-08-22T23:21:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71c721b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.741+0000 7f71c721b640 1 -- 172.21.15.52:0/1148764369 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 msgr2=0x7f71c014ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 --2- 172.21.15.52:0/1148764369 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c014ba00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009d90 tx=0x7f71b802bea0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 -- 172.21.15.52:0/1148764369 shutdown_connections 2023-08-22T23:21:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 --2- 172.21.15.52:0/1148764369 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c014ba00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 --2- 172.21.15.52:0/1148764369 >> v2:172.21.15.29:3300/0 conn(0x7f71c00add90 0x7f71c013e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 -- 172.21.15.52:0/1148764369 >> 172.21.15.52:0/1148764369 conn(0x7f71c0065ed0 msgr2=0x7f71c00b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 -- 172.21.15.52:0/1148764369 shutdown_connections 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 1 -- 172.21.15.52:0/1148764369 wait complete. 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 10 monclient: build_initial_monmap 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.742+0000 7f71c721b640 10 monclient: monmap: 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 librados: starting msgr at 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 librados: starting objecter 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 Processor -- start 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 -- start start 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 librados: setting wanted keys 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 librados: calling monclient init 2023-08-22T23:21:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient: init 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71c00add90 0x7f71c013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): picked mon.noname-b con 0x7f71c00add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c0133c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): picked mon.noname-c con 0x7f71c013e6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71c01448e0 0x7f71c0134150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): picked mon.noname-a con 0x7f71c01448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c013f290 con 0x7f71c01448e0 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c0134690 con 0x7f71c00add90 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c013eef0 con 0x7f71c013e6d0 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.743+0000 7f71c721b640 10 monclient(hunting): authenticate will time out at 1484.956853s 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71c4fb7640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71c00add90 0x7f71c013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c0133c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71c57b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71c01448e0 0x7f71c0134150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c0133c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37286/0 (socket says 172.21.15.52:37286) 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 1 -- 172.21.15.52:0/3974825690 learned_addr learned my addr 172.21.15.52:0/3974825690 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 10 monclient(hunting): get_auth_request con 0x7f71c013e6d0 auth_method 0 2023-08-22T23:21:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71bffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71c4fb7640 10 monclient(hunting): get_auth_request con 0x7f71c00add90 auth_method 0 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71c4fb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71c4fb7640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.744+0000 7f71c4fb7640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): get_auth_request con 0x7f71c01448e0 auth_method 0 2023-08-22T23:21:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:35.757 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f4deef5b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-22T23:21:35.758 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c4fb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c4fb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c4fb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.745+0000 7f71c57b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 1 -- 172.21.15.52:0/3974825690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71c00add90 msgr2=0x7f71c013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71c00add90 0x7f71c013cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 1 -- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71c01448e0 msgr2=0x7f71c0134150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71c01448e0 0x7f71c0134150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 10 monclient: _finish_hunting 0 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 1 monclient: found mon.noname-c 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.748+0000 7f71bffff640 1 -- 172.21.15.52:0/3974825690 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8009a70 con 0x7f71c013e6d0 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.750+0000 7f71bffff640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c0133c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00e9b0 tx=0x7f71ac00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.750+0000 7f71bd7fa640 10 client.?.objecter ms_handle_connect 0x7f71c013e6d0 2023-08-22T23:21:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.750+0000 7f71bd7fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac009670 con 0x7f71c013e6d0 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: dump: 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: _finish_auth 0 2023-08-22T23:21:35.763 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: _check_auth_tickets 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac00c9a0 con 0x7f71c013e6d0 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac00cd50 con 0x7f71c013e6d0 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:35.764 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71bd7fa640 10 monclient: dump: 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:35.765 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 5 monclient: authenticate success, global_id 4115 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 10 monclient: _renew_subs 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c00b6460 con 0x7f71c013e6d0 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71beffd640 4 set_mon_vals no callback set 2023-08-22T23:21:35.766 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 10 monclient: _renew_subs 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c00b5ca0 con 0x7f71c013e6d0 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.754+0000 7f71c721b640 1 librados: init done 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.755+0000 7f71c721b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.755+0000 7f71c721b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.755+0000 7f71c721b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.767 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.755+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c00b0360 con 0x7f71c013e6d0 2023-08-22T23:21:35.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.764+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f71ac005c90 con 0x7f71c013e6d0 2023-08-22T23:21:35.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.764+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f71ac013890 con 0x7f71c013e6d0 2023-08-22T23:21:35.768 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.764+0000 7f71bd7fa640 10 client.4115.objecter ms_dispatch 0x7f71c0000b90 osd_map(4..4 src has 1..4) v4 2023-08-22T23:21:35.769 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.764+0000 7f71bd7fa640 3 client.4115.objecter handle_osd_map got epochs [4,4] > 0 2023-08-22T23:21:35.769 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.764+0000 7f71bd7fa640 3 client.4115.objecter handle_osd_map decoding full epoch 4 2023-08-22T23:21:35.769 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.764+0000 7f71bd7fa640 20 client.4115.objecter dump_active .. 0 homeless 2023-08-22T23:21:35.770 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:35.769+0000 7f27b0a3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-22T23:21:35.833 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:35.832+0000 7f27b0a3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-22T23:21:35.862 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:35.861+0000 7f4deef5b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-22T23:21:35.873 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.872+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f71c00b0360 con 0x7f71c013e6d0 2023-08-22T23:21:35.873 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.872+0000 7f71bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:35.873 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.872+0000 7f71bd7fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:35.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:35.880+0000 7f27b0a3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-22T23:21:35.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.945+0000 7f71c721b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0a17913e-6533-4e19-a9c7-edd21971b8e5", "id": 2}] 2023-08-22T23:21:35.946 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.945+0000 7f71c721b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0a17913e-6533-4e19-a9c7-edd21971b8e5", "id": 2}] 2023-08-22T23:21:35.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.945+0000 7f71c721b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:35.947 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:35.945+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0a17913e-6533-4e19-a9c7-edd21971b8e5", "id": 2} v 0) v1 -- 0x7f71c013f400 con 0x7f71c013e6d0 2023-08-22T23:21:35.996 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:35.994+0000 7f27b0a3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-22T23:21:36.009 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.008+0000 7f4deef5b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-22T23:21:36.074 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.072+0000 7f71bd7fa640 1 -- 172.21.15.52:0/3974825690 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a17913e-6533-4e19-a9c7-edd21971b8e5", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f71ac021060 con 0x7f71c013e6d0 2023-08-22T23:21:36.074 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.072+0000 7f71bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0a17913e-6533-4e19-a9c7-edd21971b8e5", "id": 2}] 2023-08-22T23:21:36.074 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.072+0000 7f71bd7fa640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:36.074 INFO:teuthology.orchestra.run.smithi052.stdout:2 2023-08-22T23:21:36.075 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 10 librados: watch_flush enter 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 10 librados: watch_flush exit 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 20 client.4115.objecter shutdown clearing up homeless session... 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 10 client.4115.objecter successfully canceled tick 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 10 monclient: shutdown 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 msgr2=0x7f71c0133c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c0133c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00e9b0 tx=0x7f71ac00ee70 comp rx=0 tx=0).stop 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 shutdown_connections 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71c01448e0 0x7f71c0134150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f71c013e6d0 0x7f71c0133c10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 --2- 172.21.15.52:0/3974825690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71c00add90 0x7f71c013cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.073+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 >> 172.21.15.52:0/3974825690 conn(0x7f71c0065ed0 msgr2=0x7f71c013a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.074+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 shutdown_connections 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.074+0000 7f71c721b640 1 -- 172.21.15.52:0/3974825690 wait complete. 2023-08-22T23:21:36.077 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.074+0000 7f71c721b640 1 librados: shutdown 2023-08-22T23:21:36.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 4f849fcc-3618-4171-b806-1a2e480340f9 3 2023-08-22T23:21:36.113 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.112+0000 7f4deef5b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-22T23:21:36.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:36.145+0000 7f27b0a3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-22T23:21:36.160 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.159+0000 7f4deef5b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-22T23:21:36.217 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.215+0000 7fcf4d7c9640 10 monclient: get_monmap_and_config 2023-08-22T23:21:36.217 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.215+0000 7fcf4d7c9640 10 monclient: build_initial_monmap 2023-08-22T23:21:36.217 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.216+0000 7fcf4d7c9640 10 monclient: monmap: 2023-08-22T23:21:36.217 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:36.217 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:36.217037+0000 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:36.217037+0000 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:36.218 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:36.219 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:36.219 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.220 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.216+0000 7fcf4d7c9640 1 Processor -- start 2023-08-22T23:21:36.221 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.216+0000 7fcf4d7c9640 1 -- start start 2023-08-22T23:21:36.221 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.216+0000 7fcf4d7c9640 10 monclient: init 2023-08-22T23:21:36.221 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:36.221 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-08-22T23:21:36.221 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf480a7d50 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:36.221 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcf480a8130 0x7fcf481411e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): picked mon.noname-c con 0x7fcf480a8130 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf4814ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): picked mon.noname-b con 0x7fcf48141720 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4813ee50 con 0x7fcf480a7d50 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf480a87c0 con 0x7fcf480a8130 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.222 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480b06a0 con 0x7fcf48141720 2023-08-22T23:21:36.223 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:36.223 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.217+0000 7fcf4d7c9640 10 monclient(hunting): authenticate will time out at 1485.430150s 2023-08-22T23:21:36.223 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcf480a8130 0x7fcf481411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.223 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf46ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcf480a7d50 0x7fcf480aff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.52:51490/0 (socket says 172.21.15.52:51490) 2023-08-22T23:21:36.223 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf46ffd640 1 -- 172.21.15.52:0/3569013540 learned_addr learned my addr 172.21.15.52:0/3569013540 (peer_addr_for_me v1:172.21.15.52:0/0) 2023-08-22T23:21:36.223 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf477fe640 1 --2- 172.21.15.52:0/3569013540 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf4814ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 10 monclient(hunting): get_auth_request con 0x7fcf480a8130 auth_method 0 2023-08-22T23:21:36.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.224 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.225 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf477fe640 10 monclient(hunting): get_auth_request con 0x7fcf48141720 auth_method 0 2023-08-22T23:21:36.225 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.225 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf477fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.226 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf477fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.226 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.226 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.218+0000 7fcf467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.226 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.219+0000 7fcf467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.219+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.219+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.227 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.219+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.232 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2023-08-22T23:21:36.232 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 1 -- 172.21.15.52:0/3569013540 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 msgr2=0x7fcf4814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:36.232 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 1 --2- 172.21.15.52:0/3569013540 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf4814ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.232 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 1 -- 172.21.15.52:0/3569013540 >> v1:172.21.15.29:6789/0 conn(0x7fcf480a7d50 legacy=0x7fcf480aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.232 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 10 monclient: _finish_hunting 0 2023-08-22T23:21:36.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 1 monclient: found mon.noname-c 2023-08-22T23:21:36.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:36.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:21:36.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.220+0000 7fcf467fc640 1 -- 172.21.15.52:0/3569013540 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf480a8580 con 0x7fcf480a8130 2023-08-22T23:21:36.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.222+0000 7fcf467fc640 1 --2- 172.21.15.52:0/3569013540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a8130 0x7fcf481411e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcf380097a0 tx=0x7fcf3802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ac303a8e748539 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:36.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 1 -- 172.21.15.52:0/3569013540 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3802f050 con 0x7fcf480a8130 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: dump: 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.234 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: _finish_auth 0 2023-08-22T23:21:36.235 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:21:36.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:36.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 1 -- 172.21.15.52:0/3569013540 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38031700 con 0x7fcf480a8130 2023-08-22T23:21:36.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf4d7c9640 5 monclient: authenticate success, global_id 4123 2023-08-22T23:21:36.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf4d7c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:36.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:36.236 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf4d7c9640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf4d7c9640 4 set_mon_vals no callback set 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 1 -- 172.21.15.52:0/3569013540 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38031a90 con 0x7fcf480a8130 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.226+0000 7fcf45ffb640 10 monclient: dump: 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.238 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 10 monclient: shutdown 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/3569013540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a8130 msgr2=0x7fcf481411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/3569013540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a8130 0x7fcf481411e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcf380097a0 tx=0x7fcf3802b7a0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/3569013540 shutdown_connections 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/3569013540 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf4814ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/3569013540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a8130 0x7fcf481411e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.239 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/3569013540 >> 172.21.15.52:0/3569013540 conn(0x7fcf48065dd0 msgr2=0x7fcf480b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/3569013540 shutdown_connections 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.227+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/3569013540 wait complete. 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 10 monclient: build_initial_monmap 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 10 monclient: monmap: 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:36.240 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.241 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 1 librados: starting msgr at 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 1 librados: starting objecter 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 1 Processor -- start 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 1 -- start start 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 1 librados: setting wanted keys 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 1 librados: calling monclient init 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.228+0000 7fcf4d7c9640 10 monclient: init 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:36.242 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 0x7fcf480ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): picked mon.noname-b con 0x7fcf480a7d50 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf480a8130 0x7fcf481d95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): picked mon.noname-a con 0x7fcf480a8130 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf481d6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): picked mon.noname-c con 0x7fcf48141720 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:36.243 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf481da530 con 0x7fcf480a8130 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf481da010 con 0x7fcf480a7d50 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480acd10 con 0x7fcf48141720 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:36.244 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf4d7c9640 10 monclient(hunting): authenticate will time out at 1485.442261s 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.229+0000 7fcf46ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 0x7fcf480ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf467fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf480a8130 0x7fcf481d95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf481d6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 0x7fcf480ac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35880/0 (socket says 172.21.15.52:35880) 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 1 -- 172.21.15.52:0/1643455356 learned_addr learned my addr 172.21.15.52:0/1643455356 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): get_auth_request con 0x7fcf480a7d50 auth_method 0 2023-08-22T23:21:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): get_auth_request con 0x7fcf48141720 auth_method 0 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf467fc640 10 monclient(hunting): get_auth_request con 0x7fcf480a8130 auth_method 0 2023-08-22T23:21:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.247 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf467fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.247 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf467fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.247 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.247 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.247 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.247 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.230+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.248 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.231+0000 7fcf467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.231+0000 7fcf467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.249 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.231+0000 7fcf467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2023-08-22T23:21:36.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 1 -- 172.21.15.52:0/1643455356 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 msgr2=0x7fcf481d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf481d6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.252 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 1 -- 172.21.15.52:0/1643455356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf480a8130 msgr2=0x7fcf481d95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf480a8130 0x7fcf481d95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:21:36.253 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 1 monclient: found mon.noname-b 2023-08-22T23:21:36.254 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:36.254 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:21:36.254 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.233+0000 7fcf46ffd640 1 -- 172.21.15.52:0/1643455356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38009400 con 0x7fcf480a7d50 2023-08-22T23:21:36.254 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.235+0000 7fcf46ffd640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 0x7fcf480ac290 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000c400 tx=0x7fcf3000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.235+0000 7fcf3f7fe640 10 client.?.objecter ms_handle_connect 0x7fcf480a7d50 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.235+0000 7fcf3f7fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30005ce0 con 0x7fcf480a7d50 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: dump: 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.255 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.256 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: _finish_auth 0 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf30005e40 con 0x7fcf480a7d50 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 5 monclient: authenticate success, global_id 4126 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 10 monclient: _renew_subs 2023-08-22T23:21:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf481d9b20 con 0x7fcf480a7d50 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf300072d0 con 0x7fcf480a7d50 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf3f7fe640 10 monclient: dump: 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.258 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.259 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf457fa640 4 set_mon_vals no callback set 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 10 monclient: _renew_subs 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf481d6780 con 0x7fcf480a7d50 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.240+0000 7fcf4d7c9640 1 librados: init done 2023-08-22T23:21:36.260 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.241+0000 7fcf4d7c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:36.261 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.241+0000 7fcf4d7c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:36.261 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.241+0000 7fcf4d7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:36.261 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.241+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf480a9320 con 0x7fcf480a7d50 2023-08-22T23:21:36.261 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:36.252+0000 7f27b0a3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-22T23:21:36.298 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:36.296+0000 7f27b0a3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-22T23:21:36.309 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.307+0000 7f4deef5b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-22T23:21:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcf3001c440 con 0x7fcf480a7d50 2023-08-22T23:21:36.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fcf3001b840 con 0x7fcf480a7d50 2023-08-22T23:21:36.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 10 client.4126.objecter ms_dispatch 0x7fcf48000b90 osd_map(5..5 src has 1..5) v4 2023-08-22T23:21:36.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2023-08-22T23:21:36.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2023-08-22T23:21:36.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 20 client.4126.objecter dump_active .. 0 homeless 2023-08-22T23:21:36.352 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fcf30017070 con 0x7fcf480a7d50 2023-08-22T23:21:36.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:36.353 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.350+0000 7fcf3f7fe640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:36.425 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.423+0000 7fcf4d7c9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4f849fcc-3618-4171-b806-1a2e480340f9", "id": 3}] 2023-08-22T23:21:36.425 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.423+0000 7fcf4d7c9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4f849fcc-3618-4171-b806-1a2e480340f9", "id": 3}] 2023-08-22T23:21:36.425 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.423+0000 7fcf4d7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:36.425 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.423+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4f849fcc-3618-4171-b806-1a2e480340f9", "id": 3} v 0) v1 -- 0x7fcf480a9da0 con 0x7fcf480a7d50 2023-08-22T23:21:36.441 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:36.440+0000 7f27b0a3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-22T23:21:36.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.495+0000 7fcf3f7fe640 1 -- 172.21.15.52:0/1643455356 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f849fcc-3618-4171-b806-1a2e480340f9", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf3001b540 con 0x7fcf480a7d50 2023-08-22T23:21:36.497 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.495+0000 7fcf3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4f849fcc-3618-4171-b806-1a2e480340f9", "id": 3}] 2023-08-22T23:21:36.497 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.495+0000 7fcf3f7fe640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:36.497 INFO:teuthology.orchestra.run.smithi052.stdout:3 2023-08-22T23:21:36.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 10 librados: watch_flush enter 2023-08-22T23:21:36.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 10 librados: watch_flush exit 2023-08-22T23:21:36.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 20 client.4126.objecter shutdown clearing up homeless session... 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 10 client.4126.objecter successfully canceled tick 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 10 monclient: shutdown 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 msgr2=0x7fcf480ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 0x7fcf480ac290 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000c400 tx=0x7fcf3000c8c0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 shutdown_connections 2023-08-22T23:21:36.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcf48141720 0x7fcf481d6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf480a8130 0x7fcf481d95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 --2- 172.21.15.52:0/1643455356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcf480a7d50 0x7fcf480ac290 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 >> 172.21.15.52:0/1643455356 conn(0x7fcf48065dd0 msgr2=0x7fcf48073190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 shutdown_connections 2023-08-22T23:21:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 -- 172.21.15.52:0/1643455356 wait complete. 2023-08-22T23:21:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.496+0000 7fcf4d7c9640 1 librados: shutdown 2023-08-22T23:21:36.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 3039bfc4-a618-40c5-9111-5325782f6646 4 2023-08-22T23:21:36.612 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.611+0000 7f6524942640 10 monclient: get_monmap_and_config 2023-08-22T23:21:36.612 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.611+0000 7f6524942640 10 monclient: build_initial_monmap 2023-08-22T23:21:36.612 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.611+0000 7f6524942640 10 monclient: monmap: 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:36.612259+0000 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:36.612259+0000 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.613 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:36.614 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.615 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.611+0000 7f6524942640 1 Processor -- start 2023-08-22T23:21:36.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 1 -- start start 2023-08-22T23:21:36.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient: init 2023-08-22T23:21:36.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:36.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-08-22T23:21:36.616 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65201448e0 0x7f6520144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient(hunting): picked mon.noname-a con 0x7f65201448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:36.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65200add90 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:36.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f652013e8a0 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:36.617 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65200b12d0 con 0x7f65200add90 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65200afae0 con 0x7f652013e8a0 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65200afdc0 con 0x7f65201448e0 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.612+0000 7f6524942640 10 monclient(hunting): authenticate will time out at 1485.825597s 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.613+0000 7f651e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65201448e0 0x7f6520144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.618 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.613+0000 7f651ed9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f652013e8a0 0x7f652014ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:39050/0 (socket says 172.21.15.52:39050) 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.613+0000 7f651ed9d640 1 -- 172.21.15.52:0/3272746985 learned_addr learned my addr 172.21.15.52:0/3272746985 (peer_addr_for_me v1:172.21.15.52:0/0) 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): get_auth_request con 0x7f65201448e0 auth_method 0 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.619 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.614+0000 7f651e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.620 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.616+0000 7f651dd9b640 1 -- 172.21.15.52:0/3272746985 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278547835 0 0) 0x7f65200afae0 con 0x7f652013e8a0 2023-08-22T23:21:36.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.616+0000 7f651dd9b640 10 monclient(hunting): _negotiate 2023-08-22T23:21:36.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.616+0000 7f651dd9b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.616+0000 7f651dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.616+0000 7f651dd9b640 10 monclient(hunting): my global_id is 4129 2023-08-22T23:21:36.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.616+0000 7f651dd9b640 1 -- 172.21.15.52:0/3272746985 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f650c003520 con 0x7f652013e8a0 2023-08-22T23:21:36.621 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2023-08-22T23:21:36.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 1 -- 172.21.15.52:0/3272746985 >> v1:172.21.15.52:6789/0 conn(0x7f652013e8a0 legacy=0x7f652014ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 1 -- 172.21.15.52:0/3272746985 >> v1:172.21.15.29:6789/0 conn(0x7f65200add90 legacy=0x7f652013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.622 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 10 monclient: _finish_hunting 0 2023-08-22T23:21:36.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 1 monclient: found mon.noname-a 2023-08-22T23:21:36.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:36.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:21:36.623 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.618+0000 7f651e59c640 1 -- 172.21.15.52:0/3272746985 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65200b00a0 con 0x7f65201448e0 2023-08-22T23:21:36.625 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.624+0000 7f651e59c640 1 --2- 172.21.15.52:0/3272746985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65201448e0 0x7f6520144cc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65100097a0 tx=0x7f651002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f4874fadee7e5b server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:36.629 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 1 -- 172.21.15.52:0/3272746985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510031830 con 0x7f65201448e0 2023-08-22T23:21:36.629 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: dump: 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.630 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: _finish_auth 0 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: _check_auth_tickets 2023-08-22T23:21:36.631 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:36.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 1 -- 172.21.15.52:0/3272746985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6510031990 con 0x7f65201448e0 2023-08-22T23:21:36.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f6524942640 5 monclient: authenticate success, global_id 4137 2023-08-22T23:21:36.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f6524942640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:36.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:36.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f6524942640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:36.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f6524942640 4 set_mon_vals no callback set 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 1 -- 172.21.15.52:0/3272746985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510031d20 con 0x7f65201448e0 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.628+0000 7f651dd9b640 10 monclient: dump: 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.633 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.634 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.634 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.634 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.634 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.634 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.634 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.637 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 10 monclient: shutdown 2023-08-22T23:21:36.637 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:36.638 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 -- 172.21.15.52:0/3272746985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65201448e0 msgr2=0x7f6520144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.638 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 --2- 172.21.15.52:0/3272746985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65201448e0 0x7f6520144cc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65100097a0 tx=0x7f651002b8d0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.638 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 -- 172.21.15.52:0/3272746985 shutdown_connections 2023-08-22T23:21:36.638 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 --2- 172.21.15.52:0/3272746985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65201448e0 0x7f6520144cc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.638 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 -- 172.21.15.52:0/3272746985 >> 172.21.15.52:0/3272746985 conn(0x7f6520065ed0 msgr2=0x7f65200b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 -- 172.21.15.52:0/3272746985 shutdown_connections 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.629+0000 7f6524942640 1 -- 172.21.15.52:0/3272746985 wait complete. 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 10 monclient: build_initial_monmap 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 10 monclient: monmap: 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:36.639 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:36.640 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 1 librados: starting msgr at 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 1 librados: starting objecter 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 1 Processor -- start 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 1 -- start start 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 1 librados: setting wanted keys 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 1 librados: calling monclient init 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.630+0000 7f6524942640 10 monclient: init 2023-08-22T23:21:36.641 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65200add90 0x7f65200aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): picked mon.noname-b con 0x7f65200add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f652013e8a0 0x7f652013d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): picked mon.noname-a con 0x7f652013e8a0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 0x7f6520141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): picked mon.noname-c con 0x7f65201448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6520141150 con 0x7f652013e8a0 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6520141500 con 0x7f65200add90 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6520141670 con 0x7f65201448e0 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6524942640 10 monclient(hunting): authenticate will time out at 1485.844096s 2023-08-22T23:21:36.643 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f651e59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65200add90 0x7f65200aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f651ed9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 0x7f6520141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.631+0000 7f6517fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f652013e8a0 0x7f652013d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 0x7f6520141ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37312/0 (socket says 172.21.15.52:37312) 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 1 -- 172.21.15.52:0/4239268919 learned_addr learned my addr 172.21.15.52:0/4239268919 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): get_auth_request con 0x7f65201448e0 auth_method 0 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.644 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f6517fff640 10 monclient(hunting): get_auth_request con 0x7f652013e8a0 auth_method 0 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f6517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f6517fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f6517fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651e59c640 10 monclient(hunting): get_auth_request con 0x7f65200add90 auth_method 0 2023-08-22T23:21:36.645 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651e59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651e59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.632+0000 7f651ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.633+0000 7f651e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.633+0000 7f651e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.633+0000 7f651e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.633+0000 7f6517fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:36.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.633+0000 7f6517fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:36.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.633+0000 7f6517fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:36.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 10 monclient(hunting): handle_auth_done global_id 4124 payload 824 2023-08-22T23:21:36.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 1 -- 172.21.15.52:0/4239268919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65200add90 msgr2=0x7f65200aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.647 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65200add90 0x7f65200aca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 1 -- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f652013e8a0 msgr2=0x7f652013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f652013e8a0 0x7f652013d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 10 monclient: _finish_hunting 0 2023-08-22T23:21:36.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 1 monclient: found mon.noname-c 2023-08-22T23:21:36.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:36.648 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:36.649 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.635+0000 7f651ed9d640 1 -- 172.21.15.52:0/4239268919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6510009400 con 0x7f65201448e0 2023-08-22T23:21:36.650 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.637+0000 7f651ed9d640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 0x7f6520141ab0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f651800cbb0 tx=0x7f651800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:36.650 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.637+0000 7f6516ffd640 10 client.?.objecter ms_handle_connect 0x7f65201448e0 2023-08-22T23:21:36.650 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.637+0000 7f6516ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651800ad40 con 0x7f65201448e0 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: dump: 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.651 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.652 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: _finish_auth 0 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6518012400 con 0x7f65201448e0 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 5 monclient: authenticate success, global_id 4124 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 10 monclient: _renew_subs 2023-08-22T23:21:36.653 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652013b030 con 0x7f65201448e0 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651800aea0 con 0x7f65201448e0 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6516ffd640 10 monclient: dump: 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:36.654 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:36.655 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f651d59a640 4 set_mon_vals no callback set 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 10 monclient: _renew_subs 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65201d9e50 con 0x7f65201448e0 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.641+0000 7f6524942640 1 librados: init done 2023-08-22T23:21:36.656 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.642+0000 7f6524942640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:36.657 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.642+0000 7f6524942640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:36.657 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.642+0000 7f6524942640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:36.657 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.642+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6520066cb0 con 0x7f65201448e0 2023-08-22T23:21:36.657 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.644+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6518012a70 con 0x7f65201448e0 2023-08-22T23:21:36.729 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.728+0000 7f4deef5b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-22T23:21:36.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f6518012d20 con 0x7f65201448e0 2023-08-22T23:21:36.756 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 10 client.4124.objecter ms_dispatch 0x7f6520000b90 osd_map(6..6 src has 1..6) v4 2023-08-22T23:21:36.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 3 client.4124.objecter handle_osd_map got epochs [6,6] > 0 2023-08-22T23:21:36.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 3 client.4124.objecter handle_osd_map decoding full epoch 6 2023-08-22T23:21:36.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 20 client.4124.objecter dump_active .. 0 homeless 2023-08-22T23:21:36.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f6518021060 con 0x7f65201448e0 2023-08-22T23:21:36.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:36.757 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.755+0000 7f6516ffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.828+0000 7f6524942640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3039bfc4-a618-40c5-9111-5325782f6646", "id": 4}] 2023-08-22T23:21:36.830 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.828+0000 7f6524942640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3039bfc4-a618-40c5-9111-5325782f6646", "id": 4}] 2023-08-22T23:21:36.830 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.828+0000 7f6524942640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:36.830 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.828+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3039bfc4-a618-40c5-9111-5325782f6646", "id": 4} v 0) v1 -- 0x7f6520138df0 con 0x7f65201448e0 2023-08-22T23:21:36.878 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:36.876+0000 7f27b0a3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-22T23:21:36.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.911+0000 7f6516ffd640 1 -- 172.21.15.52:0/4239268919 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3039bfc4-a618-40c5-9111-5325782f6646", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6518017070 con 0x7f65201448e0 2023-08-22T23:21:36.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.911+0000 7f6516ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3039bfc4-a618-40c5-9111-5325782f6646", "id": 4}] 2023-08-22T23:21:36.913 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.911+0000 7f6516ffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:36.913 INFO:teuthology.orchestra.run.smithi052.stdout:4 2023-08-22T23:21:36.914 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 10 librados: watch_flush enter 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 10 librados: watch_flush exit 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 20 client.4124.objecter shutdown clearing up homeless session... 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 10 client.4124.objecter successfully canceled tick 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 10 monclient: shutdown 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 msgr2=0x7f6520141ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:36.915 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 0x7f6520141ab0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f651800cbb0 tx=0x7f651800a430 comp rx=0 tx=0).stop 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 shutdown_connections 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f65201448e0 0x7f6520141ab0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f652013e8a0 0x7f652013d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 --2- 172.21.15.52:0/4239268919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65200add90 0x7f65200aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 >> 172.21.15.52:0/4239268919 conn(0x7f6520065ed0 msgr2=0x7f652013df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 shutdown_connections 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.912+0000 7f6524942640 1 -- 172.21.15.52:0/4239268919 wait complete. 2023-08-22T23:21:36.916 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:36.913+0000 7f6524942640 1 librados: shutdown 2023-08-22T23:21:36.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 355b1e15-08a3-4837-8b46-fb14f5f272e6 5 2023-08-22T23:21:36.942 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.941+0000 7f4deef5b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-22T23:21:36.999 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:36.998+0000 7f4deef5b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-22T23:21:37.028 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.026+0000 7fb6b759e640 10 monclient: get_monmap_and_config 2023-08-22T23:21:37.028 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.026+0000 7fb6b759e640 10 monclient: build_initial_monmap 2023-08-22T23:21:37.028 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.026+0000 7fb6b759e640 10 monclient: monmap: 2023-08-22T23:21:37.029 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:37.029 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.029 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:37.027751+0000 2023-08-22T23:21:37.030 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:37.027751+0000 2023-08-22T23:21:37.030 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:37.030 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.030 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:37.031 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:37.031 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:37.031 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:37.031 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:37.031 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.027+0000 7fb6b759e640 1 Processor -- start 2023-08-22T23:21:37.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.027+0000 7fb6b759e640 1 -- start start 2023-08-22T23:21:37.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.027+0000 7fb6b759e640 10 monclient: init 2023-08-22T23:21:37.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:37.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-08-22T23:21:37.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80a7b30 0x7fb6b80a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient(hunting): picked mon.noname-b con 0x7fb6b80a7b30 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6b80aff50 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6b80b0390 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b80a9980 con 0x7fb6b80aff50 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b80a88a0 con 0x7fb6b80b0390 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80a8450 con 0x7fb6b80a7b30 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b759e640 10 monclient(hunting): authenticate will time out at 1486.241102s 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b659c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80a7b30 0x7fb6b80a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b6d9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb6b80b0390 0x7fb6b8144bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:39054/0 (socket says 172.21.15.52:39054) 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b6d9d640 1 -- 172.21.15.52:0/2222851129 learned_addr learned my addr 172.21.15.52:0/2222851129 (peer_addr_for_me v1:172.21.15.52:0/0) 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.028+0000 7fb6b5d9b640 1 --1- 172.21.15.52:0/2222851129 >> v1:172.21.15.29:6789/0 conn(0x7fb6b80aff50 0x7fb6b81444e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.52:51514/0 (socket says 172.21.15.52:51514) 2023-08-22T23:21:37.036 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): get_auth_request con 0x7fb6b80a7b30 auth_method 0 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.029+0000 7fb6b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.031+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813222554 0 0) 0x7fb6b80a88a0 con 0x7fb6b80b0390 2023-08-22T23:21:37.037 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.031+0000 7fb6b559a640 10 monclient(hunting): _negotiate 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.031+0000 7fb6b559a640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.031+0000 7fb6b559a640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.031+0000 7fb6b559a640 10 monclient(hunting): my global_id is 4135 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.031+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a0003520 con 0x7fb6b80b0390 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438757353 0 0) 0x7fb6b80a9980 con 0x7fb6b80aff50 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b559a640 10 monclient(hunting): _negotiate 2023-08-22T23:21:37.038 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b559a640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b559a640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b559a640 10 monclient(hunting): my global_id is 4146 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b80a88a0 con 0x7fb6b80aff50 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 1 -- 172.21.15.52:0/2222851129 >> v1:172.21.15.52:6789/0 conn(0x7fb6b80b0390 legacy=0x7fb6b8144bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 1 -- 172.21.15.52:0/2222851129 >> v1:172.21.15.29:6789/0 conn(0x7fb6b80aff50 legacy=0x7fb6b81444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 10 monclient: _finish_hunting 0 2023-08-22T23:21:37.039 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 1 monclient: found mon.noname-b 2023-08-22T23:21:37.040 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:37.040 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:21:37.040 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.032+0000 7fb6b659c640 1 -- 172.21.15.52:0/2222851129 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b80b09e0 con 0x7fb6b80a7b30 2023-08-22T23:21:37.042 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.034+0000 7fb6b659c640 1 --2- 172.21.15.52:0/2222851129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80a7b30 0x7fb6b80a7f10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80097d0 tx=0x7fb6a802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=207e2e81764618ea server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:37.042 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8031890 con 0x7fb6b80a7b30 2023-08-22T23:21:37.042 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.042 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:37.042 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: dump: 2023-08-22T23:21:37.042 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.043 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: _finish_auth 0 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: _check_auth_tickets 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a80319f0 con 0x7fb6b80a7b30 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 1 -- 172.21.15.52:0/2222851129 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8031d80 con 0x7fb6b80a7b30 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b759e640 5 monclient: authenticate success, global_id 4127 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b759e640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b759e640 4 set_mon_vals no callback set 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.038+0000 7fb6b559a640 10 monclient: dump: 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 10 monclient: shutdown 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 -- 172.21.15.52:0/2222851129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80a7b30 msgr2=0x7fb6b80a7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 --2- 172.21.15.52:0/2222851129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80a7b30 0x7fb6b80a7f10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80097d0 tx=0x7fb6a802b930 comp rx=0 tx=0).stop 2023-08-22T23:21:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 -- 172.21.15.52:0/2222851129 shutdown_connections 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 --2- 172.21.15.52:0/2222851129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80a7b30 0x7fb6b80a7f10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 -- 172.21.15.52:0/2222851129 >> 172.21.15.52:0/2222851129 conn(0x7fb6b80b6780 msgr2=0x7fb6b80b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 -- 172.21.15.52:0/2222851129 shutdown_connections 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.039+0000 7fb6b759e640 1 -- 172.21.15.52:0/2222851129 wait complete. 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.040+0000 7fb6b759e640 10 monclient: build_initial_monmap 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.040+0000 7fb6b759e640 10 monclient: monmap: 2023-08-22T23:21:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.040+0000 7fb6b759e640 1 librados: starting msgr at 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.040+0000 7fb6b759e640 1 librados: starting objecter 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.040+0000 7fb6b759e640 1 Processor -- start 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.040+0000 7fb6b759e640 1 -- start start 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 librados: setting wanted keys 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 librados: calling monclient init 2023-08-22T23:21:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient: init 2023-08-22T23:21:37.053 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80a7b30 0x7fb6b8143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): picked mon.noname-a con 0x7fb6b80a7b30 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80aff50 0x7fb6b80ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): picked mon.noname-c con 0x7fb6b80aff50 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6b80b0390 0x7fb6b80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): picked mon.noname-b con 0x7fb6b80b0390 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80b5c00 con 0x7fb6b80a7b30 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80abd60 con 0x7fb6b80b0390 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80ac040 con 0x7fb6b80aff50 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.041+0000 7fb6b759e640 10 monclient(hunting): authenticate will time out at 1486.254407s 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80a7b30 0x7fb6b8143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b6d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6b80b0390 0x7fb6b80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80aff50 0x7fb6b80ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80aff50 0x7fb6b80ac460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37320/0 (socket says 172.21.15.52:37320) 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 1 -- 172.21.15.52:0/1539982148 learned_addr learned my addr 172.21.15.52:0/1539982148 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 10 monclient(hunting): get_auth_request con 0x7fb6b80aff50 auth_method 0 2023-08-22T23:21:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b659c640 10 monclient(hunting): get_auth_request con 0x7fb6b80a7b30 auth_method 0 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b659c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b659c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b6d9d640 10 monclient(hunting): get_auth_request con 0x7fb6b80b0390 auth_method 0 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b6d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.042+0000 7fb6b6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.043+0000 7fb6b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 1 -- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80aff50 msgr2=0x7fb6b80ac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80aff50 0x7fb6b80ac460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.059 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 1 -- 172.21.15.52:0/1539982148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6b80b0390 msgr2=0x7fb6b80aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6b80b0390 0x7fb6b80aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 10 monclient: _finish_hunting 0 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 1 monclient: found mon.noname-a 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.045+0000 7fb6b659c640 1 -- 172.21.15.52:0/1539982148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8009400 con 0x7fb6b80a7b30 2023-08-22T23:21:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.048+0000 7fb6b659c640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80a7b30 0x7fb6b8143f20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8033000 tx=0x7fb6a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.048+0000 7fb6a6ffd640 10 client.?.objecter ms_handle_connect 0x7fb6b80a7b30 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.048+0000 7fb6a6ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a803bc30 con 0x7fb6b80a7b30 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: dump: 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.063 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.063 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: _finish_auth 0 2023-08-22T23:21:37.063 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:37.063 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:37.063 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a803bdd0 con 0x7fb6b80a7b30 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8045630 con 0x7fb6b80a7b30 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 5 monclient: authenticate success, global_id 4149 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 10 monclient: _renew_subs 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b80b5890 con 0x7fb6b80a7b30 2023-08-22T23:21:37.064 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b4d99640 4 set_mon_vals no callback set 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6a6ffd640 10 monclient: dump: 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.066 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 10 monclient: _renew_subs 2023-08-22T23:21:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b80aab70 con 0x7fb6b80a7b30 2023-08-22T23:21:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.052+0000 7fb6b759e640 1 librados: init done 2023-08-22T23:21:37.068 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.052+0000 7f4deef5b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-22T23:21:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.053+0000 7fb6b759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.053+0000 7fb6b759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.053+0000 7fb6b759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.053+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b8066cb0 con 0x7fb6b80a7b30 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.054+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb6a8045a90 con 0x7fb6b80a7b30 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.054+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fb6a80303e0 con 0x7fb6b80a7b30 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.054+0000 7fb6a6ffd640 10 client.4149.objecter ms_dispatch 0x7fb6b8000b90 osd_map(7..7 src has 1..7) v4 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.054+0000 7fb6a6ffd640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2023-08-22T23:21:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.054+0000 7fb6a6ffd640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2023-08-22T23:21:37.070 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.054+0000 7fb6a6ffd640 20 client.4149.objecter dump_active .. 0 homeless 2023-08-22T23:21:37.096 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.095+0000 7f27b0a3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-22T23:21:37.109 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.107+0000 7f4deef5b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-22T23:21:37.152 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.150+0000 7f27b0a3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-22T23:21:37.159 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.158+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fb6a8035020 con 0x7fb6b80a7b30 2023-08-22T23:21:37.159 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.158+0000 7fb6a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.159 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.158+0000 7fb6a6ffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:37.200 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.199+0000 7f27b0a3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-22T23:21:37.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.231+0000 7fb6b759e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "355b1e15-08a3-4837-8b46-fb14f5f272e6", "id": 5}] 2023-08-22T23:21:37.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.231+0000 7fb6b759e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "355b1e15-08a3-4837-8b46-fb14f5f272e6", "id": 5}] 2023-08-22T23:21:37.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.232+0000 7fb6b759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:21:37.233 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.232+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd new", "uuid": "355b1e15-08a3-4837-8b46-fb14f5f272e6", "id": 5} v 0) v1 -- 0x7fb6b80ac180 con 0x7fb6b80a7b30 2023-08-22T23:21:37.259 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.257+0000 7f27b0a3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-22T23:21:37.291 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.290+0000 7fb6a6ffd640 1 -- 172.21.15.52:0/1539982148 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "355b1e15-08a3-4837-8b46-fb14f5f272e6", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb6b80ac180 con 0x7fb6b80a7b30 2023-08-22T23:21:37.292 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.290+0000 7fb6a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "355b1e15-08a3-4837-8b46-fb14f5f272e6", "id": 5}] 2023-08-22T23:21:37.292 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.290+0000 7fb6a6ffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:37.293 INFO:teuthology.orchestra.run.smithi052.stdout:5 2023-08-22T23:21:37.294 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 10 librados: watch_flush enter 2023-08-22T23:21:37.294 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 10 librados: watch_flush exit 2023-08-22T23:21:37.295 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 20 client.4149.objecter shutdown clearing up homeless session... 2023-08-22T23:21:37.295 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 10 client.4149.objecter successfully canceled tick 2023-08-22T23:21:37.295 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 10 monclient: shutdown 2023-08-22T23:21:37.296 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:37.296 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80a7b30 msgr2=0x7fb6b8143f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.296 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80a7b30 0x7fb6b8143f20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8033000 tx=0x7fb6a80046c0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.296 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 shutdown_connections 2023-08-22T23:21:37.297 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6b80b0390 0x7fb6b80aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.297 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb6b80aff50 0x7fb6b80ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.297 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 --2- 172.21.15.52:0/1539982148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80a7b30 0x7fb6b8143f20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 >> 172.21.15.52:0/1539982148 conn(0x7fb6b80b6780 msgr2=0x7fb6b80ae390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 shutdown_connections 2023-08-22T23:21:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 -- 172.21.15.52:0/1539982148 wait complete. 2023-08-22T23:21:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.291+0000 7fb6b759e640 1 librados: shutdown 2023-08-22T23:21:37.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 45d94939-76a2-46b3-b31d-f79ee4ffc624 6 2023-08-22T23:21:37.402 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.400+0000 7f4deef5b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-22T23:21:37.447 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.445+0000 7f8fdbefd640 10 monclient: get_monmap_and_config 2023-08-22T23:21:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.445+0000 7f8fdbefd640 10 monclient: build_initial_monmap 2023-08-22T23:21:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.445+0000 7f8fdbefd640 10 monclient: monmap: 2023-08-22T23:21:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:37.448 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:37.446845+0000 2023-08-22T23:21:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:37.446845+0000 2023-08-22T23:21:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:37.449 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:37.450 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:37.451 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.446+0000 7f8fdbefd640 1 Processor -- start 2023-08-22T23:21:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.446+0000 7f8fdbefd640 1 -- start start 2023-08-22T23:21:37.452 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.446+0000 7f8fdbefd640 10 monclient: init 2023-08-22T23:21:37.455 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:37.455 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-08-22T23:21:37.456 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fd41448e0 0x7f8fd4144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.456 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd41448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:37.456 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd40add90 0x7f8fd4145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.456 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd40add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:37.457 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fd413e190 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:37.457 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40b12d0 con 0x7f8fd413e190 2023-08-22T23:21:37.457 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.457 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40afae0 con 0x7f8fd41448e0 2023-08-22T23:21:37.458 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.458 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40afdc0 con 0x7f8fd40add90 2023-08-22T23:21:37.458 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:37.458 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.447+0000 7f8fdbefd640 10 monclient(hunting): authenticate will time out at 1486.660147s 2023-08-22T23:21:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9498640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd40add90 0x7f8fd4145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9c99640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fd41448e0 0x7f8fd4144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fda49a640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f8fd413e190 0x7f8fd41410f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:39068/0 (socket says 172.21.15.52:39068) 2023-08-22T23:21:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fda49a640 1 -- 172.21.15.52:0/630082784 learned_addr learned my addr 172.21.15.52:0/630082784 (peer_addr_for_me v1:172.21.15.52:0/0) 2023-08-22T23:21:37.460 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9498640 10 monclient(hunting): get_auth_request con 0x7f8fd40add90 auth_method 0 2023-08-22T23:21:37.460 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9498640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.460 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9498640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.460 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9498640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.461 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9c99640 10 monclient(hunting): get_auth_request con 0x7f8fd41448e0 auth_method 0 2023-08-22T23:21:37.461 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.461 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9c99640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.461 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.448+0000 7f8fd9c99640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.462 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.449+0000 7f8fd9498640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.462 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.449+0000 7f8fd9498640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.462 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.449+0000 7f8fd9498640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.462 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.449+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.463 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.449+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.463 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.449+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.463 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd8c97640 1 -- 172.21.15.52:0/630082784 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991516451 0 0) 0x7f8fd40b12d0 con 0x7f8fd413e190 2023-08-22T23:21:37.463 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd8c97640 10 monclient(hunting): _negotiate 2023-08-22T23:21:37.464 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd8c97640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.464 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd8c97640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.464 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd8c97640 10 monclient(hunting): my global_id is 4141 2023-08-22T23:21:37.464 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd8c97640 1 -- 172.21.15.52:0/630082784 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc0003520 con 0x7f8fd413e190 2023-08-22T23:21:37.465 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2023-08-22T23:21:37.465 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 1 -- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd40add90 msgr2=0x7f8fd4145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:37.465 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 1 --2- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd40add90 0x7f8fd4145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.465 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 1 -- 172.21.15.52:0/630082784 >> v1:172.21.15.52:6789/0 conn(0x7f8fd413e190 legacy=0x7f8fd41410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.466 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 10 monclient: _finish_hunting 0 2023-08-22T23:21:37.466 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 1 monclient: found mon.noname-a 2023-08-22T23:21:37.466 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:37.466 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:21:37.467 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9c99640 1 -- 172.21.15.52:0/630082784 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40b00a0 con 0x7f8fd41448e0 2023-08-22T23:21:37.467 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.451+0000 7f8fd9498640 1 --2- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd40add90 0x7f8fd4145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:21:37.467 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.454+0000 7f8fd9c99640 1 --2- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd4144cc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40097a0 tx=0x7f8fc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e65d74f469f427f server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 1 -- 172.21.15.52:0/630082784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031830 con 0x7f8fd41448e0 2023-08-22T23:21:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:21:37.469 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: dump: 2023-08-22T23:21:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.470 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.471 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.471 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.471 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.471 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.472 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.472 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.472 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: _finish_auth 0 2023-08-22T23:21:37.472 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: _check_auth_tickets 2023-08-22T23:21:37.473 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:37.473 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 1 -- 172.21.15.52:0/630082784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4031990 con 0x7f8fd41448e0 2023-08-22T23:21:37.473 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fdbefd640 5 monclient: authenticate success, global_id 4152 2023-08-22T23:21:37.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fdbefd640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:37.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:37.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fdbefd640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:37.474 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fdbefd640 4 set_mon_vals no callback set 2023-08-22T23:21:37.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 1 -- 172.21.15.52:0/630082784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031d20 con 0x7f8fd41448e0 2023-08-22T23:21:37.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:21:37.475 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.458+0000 7f8fd8c97640 10 monclient: dump: 2023-08-22T23:21:37.476 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.476 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.476 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.476 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.477 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.477 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.477 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.477 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.478 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.478 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.478 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f4deef5b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-22T23:21:37.485 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 10 monclient: shutdown 2023-08-22T23:21:37.485 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 -- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 msgr2=0x7f8fd4144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd4144cc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40097a0 tx=0x7f8fc402b8d0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 -- 172.21.15.52:0/630082784 shutdown_connections 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd40add90 0x7f8fd4145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/630082784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd4144cc0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 -- 172.21.15.52:0/630082784 >> 172.21.15.52:0/630082784 conn(0x7f8fd4065ed0 msgr2=0x7f8fd40b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:37.486 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 -- 172.21.15.52:0/630082784 shutdown_connections 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.459+0000 7f8fdbefd640 1 -- 172.21.15.52:0/630082784 wait complete. 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 10 monclient: build_initial_monmap 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 10 monclient: monmap: 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:37.487 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 1 librados: starting msgr at 2023-08-22T23:21:37.488 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 1 librados: starting objecter 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 1 Processor -- start 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 1 -- start start 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 1 librados: setting wanted keys 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 1 librados: calling monclient init 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.460+0000 7f8fdbefd640 10 monclient: init 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:21:37.489 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fd40add90 0x7f8fd41435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd40add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd413e190 0x7f8fd41d8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd413e190 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd41d9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd41448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41d9b30 con 0x7f8fd41448e0 2023-08-22T23:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41d5a50 con 0x7f8fd40add90 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41d5c10 con 0x7f8fd413e190 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fdbefd640 10 monclient(hunting): authenticate will time out at 1486.674300s 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.461+0000 7f8fda49a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd41d9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.491 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9c99640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fd40add90 0x7f8fd41435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd413e190 0x7f8fd41d8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd413e190 0x7f8fd41d8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37342/0 (socket says 172.21.15.52:37342) 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 1 -- 172.21.15.52:0/2605790219 learned_addr learned my addr 172.21.15.52:0/2605790219 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 10 monclient(hunting): get_auth_request con 0x7f8fd413e190 auth_method 0 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9498640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.492 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fda49a640 10 monclient(hunting): get_auth_request con 0x7f8fd41448e0 auth_method 0 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fda49a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fda49a640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fda49a640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9c99640 10 monclient(hunting): get_auth_request con 0x7f8fd40add90 auth_method 0 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9c99640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.462+0000 7f8fd9c99640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.493 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fd9498640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fd9498640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fd9498640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fda49a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fda49a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.494 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.463+0000 7f8fda49a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 1 -- 172.21.15.52:0/2605790219 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd413e190 msgr2=0x7f8fd41d8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd413e190 0x7f8fd41d8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 1 -- 172.21.15.52:0/2605790219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 msgr2=0x7f8fd41d9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd41d9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 10 monclient: _finish_hunting 0 2023-08-22T23:21:37.495 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 1 monclient: found mon.noname-b 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.465+0000 7f8fd9c99640 1 -- 172.21.15.52:0/2605790219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd0009400 con 0x7f8fd40add90 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.468+0000 7f8fd9c99640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fd40add90 0x7f8fd41435a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4033000 tx=0x7f8fc40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.468+0000 7f8fca7fc640 10 client.?.objecter ms_handle_connect 0x7f8fd40add90 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.468+0000 7f8fca7fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc403a8f0 con 0x7f8fd40add90 2023-08-22T23:21:37.496 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: dump: 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.497 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: _finish_auth 0 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:37.498 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc403aa70 con 0x7f8fd40add90 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc403ae20 con 0x7f8fd40add90 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fdbefd640 5 monclient: authenticate success, global_id 4144 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fdbefd640 10 monclient: _renew_subs 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fdbefd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc4009400 con 0x7f8fd40add90 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fcbfff640 4 set_mon_vals no callback set 2023-08-22T23:21:37.499 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.471+0000 7f8fca7fc640 10 monclient: dump: 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 10 monclient: _renew_subs 2023-08-22T23:21:37.501 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd41d6180 con 0x7f8fd40add90 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 1 librados: init done 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.472+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4066cb0 con 0x7f8fd40add90 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.475+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8fc404d740 con 0x7f8fd40add90 2023-08-22T23:21:37.502 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.475+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f8fc4030970 con 0x7f8fd40add90 2023-08-22T23:21:37.503 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.475+0000 7f8fca7fc640 10 client.4144.objecter ms_dispatch 0x7f8fd4000b90 osd_map(8..8 src has 1..8) v4 2023-08-22T23:21:37.503 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.475+0000 7f8fca7fc640 3 client.4144.objecter handle_osd_map got epochs [8,8] > 0 2023-08-22T23:21:37.503 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.475+0000 7f8fca7fc640 3 client.4144.objecter handle_osd_map decoding full epoch 8 2023-08-22T23:21:37.503 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.475+0000 7f8fca7fc640 20 client.4144.objecter dump_active .. 0 homeless 2023-08-22T23:21:37.526 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.525+0000 7f4deef5b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-22T23:21:37.558 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.557+0000 7f27b0a3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-22T23:21:37.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.576+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f8fc4035020 con 0x7f8fd40add90 2023-08-22T23:21:37.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.576+0000 7f8fca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.577 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.576+0000 7f8fca7fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:37.613 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.611+0000 7f27b0a3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-22T23:21:37.650 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.649+0000 7f8fdbefd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "45d94939-76a2-46b3-b31d-f79ee4ffc624", "id": 6}] 2023-08-22T23:21:37.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.649+0000 7f8fdbefd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "45d94939-76a2-46b3-b31d-f79ee4ffc624", "id": 6}] 2023-08-22T23:21:37.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.649+0000 7f8fdbefd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:37.651 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.649+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd new", "uuid": "45d94939-76a2-46b3-b31d-f79ee4ffc624", "id": 6} v 0) v1 -- 0x7f8fd4071950 con 0x7f8fd40add90 2023-08-22T23:21:37.661 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.660+0000 7f27b0a3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-22T23:21:37.681 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.679+0000 7f4deef5b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-22T23:21:37.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.712+0000 7f8fca7fc640 1 -- 172.21.15.52:0/2605790219 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45d94939-76a2-46b3-b31d-f79ee4ffc624", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8fc4030c00 con 0x7f8fd40add90 2023-08-22T23:21:37.713 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.712+0000 7f8fca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "45d94939-76a2-46b3-b31d-f79ee4ffc624", "id": 6}] 2023-08-22T23:21:37.714 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.712+0000 7f8fca7fc640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:37.714 INFO:teuthology.orchestra.run.smithi052.stdout:6 2023-08-22T23:21:37.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 10 librados: watch_flush enter 2023-08-22T23:21:37.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 10 librados: watch_flush exit 2023-08-22T23:21:37.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 20 client.4144.objecter shutdown clearing up homeless session... 2023-08-22T23:21:37.715 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 10 client.4144.objecter successfully canceled tick 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 10 monclient: shutdown 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fd40add90 msgr2=0x7f8fd41435a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fd40add90 0x7f8fd41435a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4033000 tx=0x7f8fc40046c0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 shutdown_connections 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd41448e0 0x7f8fd41d9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.716 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fd413e190 0x7f8fd41d8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 --2- 172.21.15.52:0/2605790219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fd40add90 0x7f8fd41435a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 >> 172.21.15.52:0/2605790219 conn(0x7f8fd4065ed0 msgr2=0x7f8fd4144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:37.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 shutdown_connections 2023-08-22T23:21:37.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 -- 172.21.15.52:0/2605790219 wait complete. 2023-08-22T23:21:37.717 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.713+0000 7f8fdbefd640 1 librados: shutdown 2023-08-22T23:21:37.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new e79b6fdf-58b5-440a-96b0-da1065376e86 7 2023-08-22T23:21:37.728 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.728+0000 7f4deef5b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-22T23:21:37.786 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.785+0000 7f4deef5b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-22T23:21:37.813 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.812+0000 7f27b0a3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-22T23:21:37.829 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.828+0000 7f7c0e8a9640 10 monclient: get_monmap_and_config 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.828+0000 7f7c0e8a9640 10 monclient: build_initial_monmap 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.828+0000 7f7c0e8a9640 10 monclient: monmap: 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:21:37.829557+0000 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:21:37.829557+0000 2023-08-22T23:21:37.830 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:37.831 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.833 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 1 Processor -- start 2023-08-22T23:21:37.833 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 1 -- start start 2023-08-22T23:21:37.833 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 10 monclient: init 2023-08-22T23:21:37.833 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f7c081448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.829+0000 7f7c0e8a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c080add90 0x7f7c08145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f7c080add90 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c0813e190 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:37.834 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080b12d0 con 0x7f7c0813e190 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c080afae0 con 0x7f7c080add90 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c080afdc0 con 0x7f7c081448e0 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c0e8a9640 10 monclient(hunting): authenticate will time out at 1487.042934s 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.835 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.830+0000 7f7c077fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c080add90 0x7f7c08145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37358/0 (socket says 172.21.15.52:37358) 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 1 -- 172.21.15.52:0/2377733204 learned_addr learned my addr 172.21.15.52:0/2377733204 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): get_auth_request con 0x7f7c081448e0 auth_method 0 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.836 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c077fe640 10 monclient(hunting): get_auth_request con 0x7f7c080add90 auth_method 0 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c077fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c077fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.831+0000 7f7c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.832+0000 7f7c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.837 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.832+0000 7f7c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.838 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.832+0000 7f7c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.838 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2023-08-22T23:21:37.838 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 1 -- 172.21.15.52:0/2377733204 >> v2:172.21.15.29:3300/0 conn(0x7f7c080add90 msgr2=0x7f7c08145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 1 --2- 172.21.15.52:0/2377733204 >> v2:172.21.15.29:3300/0 conn(0x7f7c080add90 0x7f7c08145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 1 -- 172.21.15.52:0/2377733204 >> v1:172.21.15.29:6789/0 conn(0x7f7c0813e190 legacy=0x7f7c081410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 10 monclient: _finish_hunting 0 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 1 monclient: found mon.noname-b 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.833+0000 7f7c07fff640 1 -- 172.21.15.52:0/2377733204 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c080b00a0 con 0x7f7c081448e0 2023-08-22T23:21:37.839 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.835+0000 7f7c07fff640 1 --2- 172.21.15.52:0/2377733204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08144cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8009870 tx=0x7f7bf802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ab73f7d9e7641e4 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:37.840 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:37.836+0000 7f4deef5b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-22T23:21:37.840 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 1 -- 172.21.15.52:0/2377733204 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8030840 con 0x7f7c081448e0 2023-08-22T23:21:37.840 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: dump: 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.841 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: _finish_auth 0 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:37.842 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c0e8a9640 5 monclient: authenticate success, global_id 4139 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 1 -- 172.21.15.52:0/2377733204 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf80309a0 con 0x7f7c081448e0 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c0e8a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c0e8a9640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 1 -- 172.21.15.52:0/2377733204 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8030d30 con 0x7f7c081448e0 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c0e8a9640 4 set_mon_vals no callback set 2023-08-22T23:21:37.843 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.839+0000 7f7c06ffd640 10 monclient: dump: 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.844 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.845 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.845 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.845 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.845 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.847 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 10 monclient: shutdown 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/2377733204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 msgr2=0x7f7c08144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/2377733204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08144cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8009870 tx=0x7f7bf802b980 comp rx=0 tx=0).stop 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/2377733204 shutdown_connections 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/2377733204 >> v2:172.21.15.29:3300/0 conn(0x7f7c080add90 0x7f7c08145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/2377733204 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08144cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.848 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/2377733204 >> 172.21.15.52:0/2377733204 conn(0x7f7c08065ed0 msgr2=0x7f7c080b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/2377733204 shutdown_connections 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.840+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/2377733204 wait complete. 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 10 monclient: build_initial_monmap 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 10 monclient: monmap: 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:37.849 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.850 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 1 librados: starting msgr at 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 1 librados: starting objecter 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 1 Processor -- start 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 1 -- start start 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 1 librados: setting wanted keys 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 1 librados: calling monclient init 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.841+0000 7f7c0e8a9640 10 monclient: init 2023-08-22T23:21:37.851 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c080add90 0x7f7c08143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f7c080add90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c0813e190 0x7f7c08140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0813e190 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): picked mon.noname-c con 0x7f7c081448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:37.852 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c081414b0 con 0x7f7c080add90 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c081417d0 con 0x7f7c0813e190 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c08140b00 con 0x7f7c081448e0 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0e8a9640 10 monclient(hunting): authenticate will time out at 1487.055054s 2023-08-22T23:21:37.853 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c0ce46640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c077fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c0813e190 0x7f7c08140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.842+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c080add90 0x7f7c08143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08135d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.52:37362/0 (socket says 172.21.15.52:37362) 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 1 -- 172.21.15.52:0/265802994 learned_addr learned my addr 172.21.15.52:0/265802994 (peer_addr_for_me v2:172.21.15.52:0/0) 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): get_auth_request con 0x7f7c081448e0 auth_method 0 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.854 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): get_auth_request con 0x7f7c0813e190 auth_method 0 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): get_auth_request con 0x7f7c080add90 auth_method 0 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:37.855 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c0ce46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.856 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.843+0000 7f7c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 1 -- 172.21.15.52:0/265802994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c0813e190 msgr2=0x7f7c08140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c0813e190 0x7f7c08140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 1 -- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c080add90 msgr2=0x7f7c08143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c080add90 0x7f7c08143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:37.857 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 10 monclient: _finish_hunting 0 2023-08-22T23:21:37.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 1 monclient: found mon.noname-c 2023-08-22T23:21:37.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:37.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:37.858 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.846+0000 7f7c0ce46640 1 -- 172.21.15.52:0/265802994 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf80094d0 con 0x7f7c081448e0 2023-08-22T23:21:37.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.848+0000 7f7c0ce46640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08135d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c7a0 tx=0x7f7c0000cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:37.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.848+0000 7f7c04ff9640 10 client.?.objecter ms_handle_connect 0x7f7c081448e0 2023-08-22T23:21:37.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.848+0000 7f7c04ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:37.859 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00003b70 con 0x7f7c081448e0 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: dump: 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: _finish_auth 0 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:21:37.861 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00003cf0 con 0x7f7c081448e0 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00003ea0 con 0x7f7c081448e0 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 5 monclient: authenticate success, global_id 4142 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 10 monclient: _renew_subs 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:37.862 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c08140d80 con 0x7f7c081448e0 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c067fc640 4 set_mon_vals no callback set 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c04ff9640 10 monclient: dump: 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.863 INFO:teuthology.orchestra.run.smithi052.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:37.864 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 10 monclient: _renew_subs 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c08136430 con 0x7f7c081448e0 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.852+0000 7f7c0e8a9640 1 librados: init done 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.853+0000 7f7c0e8a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.853+0000 7f7c0e8a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.853+0000 7f7c0e8a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:37.865 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.853+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c081d93b0 con 0x7f7c081448e0 2023-08-22T23:21:37.866 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.860+0000 7f27b0a3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-22T23:21:37.916 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.915+0000 7f27b0a3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-22T23:21:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.957+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7c000050d0 con 0x7f7c081448e0 2023-08-22T23:21:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7c00025a90 con 0x7f7c081448e0 2023-08-22T23:21:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 10 client.4142.objecter ms_dispatch 0x7f7c08000b90 osd_map(9..9 src has 1..9) v4 2023-08-22T23:21:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 3 client.4142.objecter handle_osd_map got epochs [9,9] > 0 2023-08-22T23:21:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 3 client.4142.objecter handle_osd_map decoding full epoch 9 2023-08-22T23:21:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 20 client.4142.objecter dump_active .. 0 homeless 2023-08-22T23:21:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f7c081d93b0 con 0x7f7c081448e0 2023-08-22T23:21:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:37.963 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:37.961+0000 7f7c04ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:37.964 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-08-22T23:21:37.962+0000 7f27b0a3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-22T23:21:38.034 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.033+0000 7f7c0e8a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e79b6fdf-58b5-440a-96b0-da1065376e86", "id": 7}] 2023-08-22T23:21:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.033+0000 7f7c0e8a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e79b6fdf-58b5-440a-96b0-da1065376e86", "id": 7}] 2023-08-22T23:21:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.033+0000 7f7c0e8a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.033+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e79b6fdf-58b5-440a-96b0-da1065376e86", "id": 7} v 0) v1 -- 0x7f7c080b0330 con 0x7f7c081448e0 2023-08-22T23:21:38.231 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.230+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mgrmap(e 2) v1 ==== 44122+0+0 (secure 0 0 0) 0x7f7c00025c50 con 0x7f7c081448e0 2023-08-22T23:21:38.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.631+0000 7f7c04ff9640 1 -- 172.21.15.52:0/265802994 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e79b6fdf-58b5-440a-96b0-da1065376e86", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7c000054c0 con 0x7f7c081448e0 2023-08-22T23:21:38.632 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.631+0000 7f7c04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e79b6fdf-58b5-440a-96b0-da1065376e86", "id": 7}] 2023-08-22T23:21:38.633 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.631+0000 7f7c04ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:38.633 INFO:teuthology.orchestra.run.smithi052.stdout:7 2023-08-22T23:21:38.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 10 librados: watch_flush enter 2023-08-22T23:21:38.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 10 librados: watch_flush exit 2023-08-22T23:21:38.634 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 20 client.4142.objecter shutdown clearing up homeless session... 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 10 client.4142.objecter successfully canceled tick 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 10 monclient: shutdown 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 msgr2=0x7f7c08135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08135d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c7a0 tx=0x7f7c0000cc60 comp rx=0 tx=0).stop 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 shutdown_connections 2023-08-22T23:21:38.635 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7c081448e0 0x7f7c08135d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c0813e190 0x7f7c08140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 --2- 172.21.15.52:0/265802994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c080add90 0x7f7c08143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 >> 172.21.15.52:0/265802994 conn(0x7f7c08065ed0 msgr2=0x7f7c08144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 shutdown_connections 2023-08-22T23:21:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 -- 172.21.15.52:0/265802994 wait complete. 2023-08-22T23:21:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:2023-08-22T23:21:38.632+0000 7f7c0e8a9640 1 librados: shutdown 2023-08-22T23:21:38.643 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.644 INFO:tasks.ceph.osd.0:Restarting daemon 2023-08-22T23:21:38.644 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-08-22T23:21:38.646 INFO:tasks.ceph.osd.0:Started 2023-08-22T23:21:38.646 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.647 INFO:tasks.ceph.osd.1:Restarting daemon 2023-08-22T23:21:38.647 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-08-22T23:21:38.649 INFO:tasks.ceph.osd.1:Started 2023-08-22T23:21:38.649 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.649 INFO:tasks.ceph.osd.2:Restarting daemon 2023-08-22T23:21:38.650 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-08-22T23:21:38.651 INFO:tasks.ceph.osd.2:Started 2023-08-22T23:21:38.652 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.652 INFO:tasks.ceph.osd.3:Restarting daemon 2023-08-22T23:21:38.653 DEBUG:teuthology.orchestra.run.smithi029:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-08-22T23:21:38.655 INFO:tasks.ceph.osd.3:Started 2023-08-22T23:21:38.655 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.655 INFO:tasks.ceph.osd.4:Restarting daemon 2023-08-22T23:21:38.656 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-08-22T23:21:38.686 INFO:tasks.ceph.osd.4:Started 2023-08-22T23:21:38.686 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.687 INFO:tasks.ceph.osd.5:Restarting daemon 2023-08-22T23:21:38.687 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-08-22T23:21:38.689 INFO:tasks.ceph.osd.5:Started 2023-08-22T23:21:38.689 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.690 INFO:tasks.ceph.osd.6:Restarting daemon 2023-08-22T23:21:38.690 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-08-22T23:21:38.692 INFO:tasks.ceph.osd.6:Started 2023-08-22T23:21:38.692 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-08-22T23:21:38.692 INFO:tasks.ceph.osd.7:Restarting daemon 2023-08-22T23:21:38.693 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-08-22T23:21:38.695 INFO:tasks.ceph.osd.7:Started 2023-08-22T23:21:38.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-22T23:21:38.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.851+0000 7f7a7036b640 10 monclient: get_monmap_and_config 2023-08-22T23:21:38.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.851+0000 7f7a7036b640 10 monclient: build_initial_monmap 2023-08-22T23:21:38.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.851+0000 7f7a7036b640 10 monclient: monmap: 2023-08-22T23:21:38.853 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:21:38.852884+0000 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:21:38.852884+0000 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:38.855 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.852+0000 7f7a7036b640 1 Processor -- start 2023-08-22T23:21:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.852+0000 7f7a7036b640 1 -- start start 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.852+0000 7f7a7036b640 10 monclient: init 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a680a8210 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f7a680b01e0 0x7f7a68144690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a680b01e0 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a680b0600 addr v1:172.21.15.52:6789/0 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680a7770 con 0x7f7a680a8210 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a681452e0 con 0x7f7a680b0600 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680a8d30 con 0x7f7a680b01e0 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.853+0000 7f7a7036b640 10 monclient(hunting): authenticate will time out at 1463.103938s 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f7a680b01e0 0x7f7a68144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f7a680b01e0 0x7f7a68144690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:41732/0 (socket says 172.21.15.29:41732) 2023-08-22T23:21:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 1 -- 172.21.15.29:0/1465468475 learned_addr learned my addr 172.21.15.29:0/1465468475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 10 monclient(hunting): get_auth_request con 0x7f7a680b01e0 auth_method 0 2023-08-22T23:21:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.854+0000 7f7a6d906640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.855+0000 7f7a6d906640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.855+0000 7f7a6d906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.855+0000 7f7a6d906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 10 monclient(hunting): handle_auth_done global_id 4162 payload 274 2023-08-22T23:21:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 1 -- 172.21.15.29:0/1465468475 >> v1:172.21.15.52:6789/0 conn(0x7f7a680b0600 legacy=0x7f7a68144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 1 -- 172.21.15.29:0/1465468475 >> v1:172.21.15.29:6789/0 conn(0x7f7a680a8210 legacy=0x7f7a680a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 10 monclient: _finish_hunting 0 2023-08-22T23:21:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 1 monclient: found mon.noname-c 2023-08-22T23:21:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:21:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.858+0000 7f7a6d906640 1 -- 172.21.15.29:0/1465468475 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680b2c00 con 0x7f7a680b01e0 2023-08-22T23:21:38.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.865+0000 7f7a6d906640 1 --2- 172.21.15.29:0/1465468475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68144690 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009870 tx=0x7f7a5802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c685cbced9faf150 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:38.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 1 -- 172.21.15.29:0/1465468475 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58030860 con 0x7f7a680b01e0 2023-08-22T23:21:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 10 monclient: dump: 2023-08-22T23:21:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:38.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 10 monclient: _finish_auth 0 2023-08-22T23:21:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 10 monclient: _check_auth_tickets 2023-08-22T23:21:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a6d105640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a7036b640 5 monclient: authenticate success, global_id 4162 2023-08-22T23:21:38.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.868+0000 7f7a7036b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a6d105640 1 -- 172.21.15.29:0/1465468475 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a580309c0 con 0x7f7a680b01e0 2023-08-22T23:21:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a6d105640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a6d105640 1 -- 172.21.15.29:0/1465468475 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58030d50 con 0x7f7a680b01e0 2023-08-22T23:21:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a7036b640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a7036b640 4 set_mon_vals no callback set 2023-08-22T23:21:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a6d105640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a6d105640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.870+0000 7f7a6d105640 10 monclient: dump: 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:38.876 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 10 monclient: shutdown 2023-08-22T23:21:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 -- 172.21.15.29:0/1465468475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 msgr2=0x7f7a68144690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 --2- 172.21.15.29:0/1465468475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68144690 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009870 tx=0x7f7a5802b9a0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 -- 172.21.15.29:0/1465468475 shutdown_connections 2023-08-22T23:21:38.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 --2- 172.21.15.29:0/1465468475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68144690 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 -- 172.21.15.29:0/1465468475 >> 172.21.15.29:0/1465468475 conn(0x7f7a680b6a10 msgr2=0x7f7a680b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:38.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 -- 172.21.15.29:0/1465468475 shutdown_connections 2023-08-22T23:21:38.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.871+0000 7f7a7036b640 1 -- 172.21.15.29:0/1465468475 wait complete. 2023-08-22T23:21:38.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.872+0000 7f7a7036b640 10 monclient: build_initial_monmap 2023-08-22T23:21:38.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.872+0000 7f7a7036b640 10 monclient: monmap: 2023-08-22T23:21:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:21:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:21:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:21:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 1 librados: starting msgr at 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 1 librados: starting objecter 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 1 Processor -- start 2023-08-22T23:21:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 1 -- start start 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 1 librados: setting wanted keys 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 1 librados: calling monclient init 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.873+0000 7f7a7036b640 10 monclient: init 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a680a8210 0x7f7a68143880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a680a8210 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68140820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a680b01e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 0x7f7a6813f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a680b0600 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680aae70 con 0x7f7a680a8210 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680ab080 con 0x7f7a680b01e0 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6813ef30 con 0x7f7a680b0600 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.874+0000 7f7a7036b640 10 monclient(hunting): authenticate will time out at 1463.125182s 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 0x7f7a6813f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6d906640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68140820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 0x7f7a6813f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:56412/0 (socket says 172.21.15.29:56412) 2023-08-22T23:21:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 1 -- 172.21.15.29:0/4096068340 learned_addr learned my addr 172.21.15.29:0/4096068340 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 10 monclient(hunting): get_auth_request con 0x7f7a680b0600 auth_method 0 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.875+0000 7f7a6e908640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): get_auth_request con 0x7f7a680b01e0 auth_method 0 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6e908640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6e908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6e908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.876+0000 7f7a6d906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 10 monclient(hunting): handle_auth_done global_id 4157 payload 824 2023-08-22T23:21:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 1 -- 172.21.15.29:0/4096068340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 msgr2=0x7f7a68140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68140820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 1 -- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a680a8210 msgr2=0x7f7a68143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a680a8210 0x7f7a68143880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:38.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 10 monclient: _finish_hunting 0 2023-08-22T23:21:38.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 1 monclient: found mon.noname-c 2023-08-22T23:21:38.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:38.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:38.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.879+0000 7f7a6e908640 1 -- 172.21.15.29:0/4096068340 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a580094d0 con 0x7f7a680b0600 2023-08-22T23:21:38.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.893+0000 7f7a6e908640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 0x7f7a6813f450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c6b0 tx=0x7f7a6000cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:38.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.893+0000 7f7a5effd640 10 client.?.objecter ms_handle_connect 0x7f7a680b0600 2023-08-22T23:21:38.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.893+0000 7f7a5effd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:38.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60003b70 con 0x7f7a680b0600 2023-08-22T23:21:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 10 monclient: dump: 2023-08-22T23:21:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:38.927 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 10 monclient: _finish_auth 0 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a7036b640 5 monclient: authenticate success, global_id 4157 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a7036b640 10 monclient: _renew_subs 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.922+0000 7f7a7036b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.923+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a681444b0 con 0x7f7a680b0600 2023-08-22T23:21:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.923+0000 7f7a7036b640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.923+0000 7f7a7036b640 10 monclient: _renew_subs 2023-08-22T23:21:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.923+0000 7f7a7036b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.923+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6813f990 con 0x7f7a680b0600 2023-08-22T23:21:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.923+0000 7f7a7036b640 1 librados: init done 2023-08-22T23:21:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.924+0000 7f7a7036b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.924+0000 7f7a7036b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.924+0000 7f7a7036b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.924+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a68066cb0 con 0x7f7a680b0600 2023-08-22T23:21:38.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a60001790 con 0x7f7a680b0600 2023-08-22T23:21:38.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a5effd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:38.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a600050d0 con 0x7f7a680b0600 2023-08-22T23:21:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a5effd640 10 monclient: dump: 2023-08-22T23:21:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:21:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:38.951 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:21:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.947+0000 7f7a6c904640 4 set_mon_vals no callback set 2023-08-22T23:21:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:38.948+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 2) v1 ==== 44122+0+0 (secure 0 0 0) 0x7f7a600052f0 con 0x7f7a680b0600 2023-08-22T23:21:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.111+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7a60005780 con 0x7f7a680b0600 2023-08-22T23:21:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.111+0000 7f7a5effd640 10 client.4157.objecter ms_dispatch 0x7f7a68000b90 osd_map(10..10 src has 1..10) v4 2023-08-22T23:21:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.111+0000 7f7a5effd640 3 client.4157.objecter handle_osd_map got epochs [10,10] > 0 2023-08-22T23:21:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.111+0000 7f7a5effd640 3 client.4157.objecter handle_osd_map decoding full epoch 10 2023-08-22T23:21:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.111+0000 7f7a5effd640 20 client.4157.objecter dump_active .. 0 homeless 2023-08-22T23:21:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.128+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f7a600055a0 con 0x7f7a680b0600 2023-08-22T23:21:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.128+0000 7f7a5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.128+0000 7f7a5effd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:39.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.252+0000 7f7a7036b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:39.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.252+0000 7f7a7036b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:39.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.252+0000 7f7a7036b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:39.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.252+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a6813f100 con 0x7f7a680b0600 2023-08-22T23:21:39.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.332+0000 7f7a5effd640 1 -- 172.21.15.29:0/4096068340 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7a6001b400 con 0x7f7a680b0600 2023-08-22T23:21:39.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.332+0000 7f7a5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:39.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.332+0000 7f7a5effd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:39.335 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:21:39.335 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:21:38.340112+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:21:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 10 librados: watch_flush enter 2023-08-22T23:21:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 10 librados: watch_flush exit 2023-08-22T23:21:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 20 client.4157.objecter shutdown clearing up homeless session... 2023-08-22T23:21:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 10 client.4157.objecter successfully canceled tick 2023-08-22T23:21:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 10 monclient: shutdown 2023-08-22T23:21:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 msgr2=0x7f7a6813f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 0x7f7a6813f450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c6b0 tx=0x7f7a6000cb70 comp rx=0 tx=0).stop 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 shutdown_connections 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7a680b0600 0x7f7a6813f450 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a680b01e0 0x7f7a68140820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 --2- 172.21.15.29:0/4096068340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a680a8210 0x7f7a68143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 >> 172.21.15.29:0/4096068340 conn(0x7f7a680b6a10 msgr2=0x7f7a680b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 shutdown_connections 2023-08-22T23:21:39.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 -- 172.21.15.29:0/4096068340 wait complete. 2023-08-22T23:21:39.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:21:39.334+0000 7f7a7036b640 1 librados: shutdown 2023-08-22T23:21:39.351 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-08-22T23:21:39.351 INFO:tasks.ceph:Waiting for OSDs to come up 2023-08-22T23:21:39.878 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:39.877+0000 7f4d8f124640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-08-22T23:21:40.134 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:21:40.133+0000 7f4d8f124640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-08-22T23:21:40.156 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:21:40.305 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.303+0000 7efc06495640 10 monclient: get_monmap_and_config 2023-08-22T23:21:40.305 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.303+0000 7efc06495640 10 monclient: build_initial_monmap 2023-08-22T23:21:40.305 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.303+0000 7efc06495640 10 monclient: monmap: 2023-08-22T23:21:40.305 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:21:40.305 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:40.305 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:21:40.304611+0000 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:21:40.304611+0000 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:40.306 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:40.307 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:40.307 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.304+0000 7efc06495640 1 Processor -- start 2023-08-22T23:21:40.307 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.304+0000 7efc06495640 1 -- start start 2023-08-22T23:21:40.307 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.304+0000 7efc06495640 10 monclient: init 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7efc0013e800 0x7efc000a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): picked mon.noname-c con 0x7efc0013e800 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efc000a88b0 0x7efc000a8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): picked mon.noname-a con 0x7efc000a88b0 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:40.308 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc001414a0 addr v1:172.21.15.29:6789/0 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc000a8dd0 con 0x7efc001414a0 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc000a8740 con 0x7efc000a88b0 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc00141fe0 con 0x7efc0013e800 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:40.309 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.305+0000 7efc06495640 10 monclient(hunting): authenticate will time out at 1464.556008s 2023-08-22T23:21:40.330 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.328+0000 7efc04a32640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc001414a0 0x7efc001418d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39812/0 (socket says 172.21.15.29:39812) 2023-08-22T23:21:40.330 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.328+0000 7efc04a32640 1 -- 172.21.15.29:0/355527187 learned_addr learned my addr 172.21.15.29:0/355527187 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:21:40.331 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.328+0000 7efbfffff640 1 --2- 172.21.15.29:0/355527187 >> v2:172.21.15.52:3300/0 conn(0x7efc0013e800 0x7efc000a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:40.331 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.330+0000 7efbff7fe640 1 --2- 172.21.15.29:0/355527187 >> v2:172.21.15.29:3300/0 conn(0x7efc000a88b0 0x7efc000a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:40.332 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.331+0000 7efbfffff640 10 monclient(hunting): get_auth_request con 0x7efc0013e800 auth_method 0 2023-08-22T23:21:40.333 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.331+0000 7efbfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:40.333 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.331+0000 7efbfffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:40.333 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.331+0000 7efbfffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:40.335 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.333+0000 7efbfeffd640 1 -- 172.21.15.29:0/355527187 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133053508 0 0) 0x7efc000a8dd0 con 0x7efc001414a0 2023-08-22T23:21:40.335 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.333+0000 7efbfeffd640 10 monclient(hunting): _negotiate 2023-08-22T23:21:40.335 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.333+0000 7efbfeffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:40.335 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.333+0000 7efbfeffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:40.336 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.333+0000 7efbfeffd640 10 monclient(hunting): my global_id is 4167 2023-08-22T23:21:40.336 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.333+0000 7efbfeffd640 1 -- 172.21.15.29:0/355527187 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe8003520 con 0x7efc001414a0 2023-08-22T23:21:40.336 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.334+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:40.336 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.334+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:40.336 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.334+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:40.339 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 10 monclient(hunting): handle_auth_done global_id 4168 payload 274 2023-08-22T23:21:40.339 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 1 -- 172.21.15.29:0/355527187 >> v2:172.21.15.29:3300/0 conn(0x7efc000a88b0 msgr2=0x7efc000a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:40.339 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 1 --2- 172.21.15.29:0/355527187 >> v2:172.21.15.29:3300/0 conn(0x7efc000a88b0 0x7efc000a8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:40.339 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 1 -- 172.21.15.29:0/355527187 >> v1:172.21.15.29:6789/0 conn(0x7efc001414a0 legacy=0x7efc001418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:40.339 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 10 monclient: _finish_hunting 0 2023-08-22T23:21:40.340 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 1 monclient: found mon.noname-c 2023-08-22T23:21:40.340 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:40.340 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:21:40.340 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.336+0000 7efbfffff640 1 -- 172.21.15.29:0/355527187 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0013ebe0 con 0x7efc0013e800 2023-08-22T23:21:40.341 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.339+0000 7efbfffff640 1 --2- 172.21.15.29:0/355527187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc000a7cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc000b7540 tx=0x7efbf002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e72870c7036a2dc5 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:40.344 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 1 -- 172.21.15.29:0/355527187 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf00317e0 con 0x7efc0013e800 2023-08-22T23:21:40.344 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:40.344 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:40.344 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 10 monclient: dump: 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:40.345 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 10 monclient: _finish_auth 0 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 10 monclient: _check_auth_tickets 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.342+0000 7efbfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.343+0000 7efc06495640 5 monclient: authenticate success, global_id 4168 2023-08-22T23:21:40.346 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.343+0000 7efc06495640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:40.347 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efbfeffd640 1 -- 172.21.15.29:0/355527187 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf0031940 con 0x7efc0013e800 2023-08-22T23:21:40.347 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efbfeffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efbfeffd640 1 -- 172.21.15.29:0/355527187 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0031cf0 con 0x7efc0013e800 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efc06495640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efc06495640 4 set_mon_vals no callback set 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efbfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efbfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.345+0000 7efbfeffd640 10 monclient: dump: 2023-08-22T23:21:40.348 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:40.349 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 10 monclient: shutdown 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 -- 172.21.15.29:0/355527187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 msgr2=0x7efc000a7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 --2- 172.21.15.29:0/355527187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc000a7cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc000b7540 tx=0x7efbf002b880 comp rx=0 tx=0).stop 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 -- 172.21.15.29:0/355527187 shutdown_connections 2023-08-22T23:21:40.350 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 --2- 172.21.15.29:0/355527187 >> v2:172.21.15.29:3300/0 conn(0x7efc000a88b0 0x7efc000a8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:40.351 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 --2- 172.21.15.29:0/355527187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc000a7cc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:40.351 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 -- 172.21.15.29:0/355527187 >> 172.21.15.29:0/355527187 conn(0x7efc000b69c0 msgr2=0x7efc000b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:40.351 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 -- 172.21.15.29:0/355527187 shutdown_connections 2023-08-22T23:21:40.351 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.346+0000 7efc06495640 1 -- 172.21.15.29:0/355527187 wait complete. 2023-08-22T23:21:40.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.347+0000 7efc06495640 10 monclient: build_initial_monmap 2023-08-22T23:21:40.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.347+0000 7efc06495640 10 monclient: monmap: 2023-08-22T23:21:40.352 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:21:40.352 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:40.352 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:21:40.352 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:40.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.347+0000 7efc06495640 1 librados: starting msgr at 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.347+0000 7efc06495640 1 librados: starting objecter 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.348+0000 7efc06495640 1 Processor -- start 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.348+0000 7efc06495640 1 -- start start 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.348+0000 7efc06495640 1 librados: setting wanted keys 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.348+0000 7efc06495640 1 librados: calling monclient init 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.348+0000 7efc06495640 10 monclient: init 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:40.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc0013ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): picked mon.noname-b con 0x7efc0013e800 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 0x7efc001dd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): picked mon.noname-c con 0x7efc001414a0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc0013a9d0 0x7efc001ddf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): picked mon.noname-a con 0x7efc0013a9d0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:40.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00145570 con 0x7efc0013a9d0 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00145050 con 0x7efc0013e800 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc00143d00 con 0x7efc001414a0 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.349+0000 7efc06495640 10 monclient(hunting): authenticate will time out at 1464.599767s 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 0x7efc001dd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:40.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbfffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc0013ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 0x7efc001dd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:56420/0 (socket says 172.21.15.29:56420) 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 1 -- 172.21.15.29:0/3992944436 learned_addr learned my addr 172.21.15.29:0/3992944436 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 10 monclient(hunting): get_auth_request con 0x7efc001414a0 auth_method 0 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:40.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbfffff640 10 monclient(hunting): get_auth_request con 0x7efc0013e800 auth_method 0 2023-08-22T23:21:40.358 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:40.358 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbfffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:40.358 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.350+0000 7efbfffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:40.358 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.351+0000 7efbff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.351+0000 7efbff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.351+0000 7efbff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.351+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.351+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.351+0000 7efbfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 1 -- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 msgr2=0x7efc0013ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:40.359 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc0013ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 1 -- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc0013a9d0 msgr2=0x7efc001ddf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc0013a9d0 0x7efc001ddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 1 monclient: found mon.noname-c 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:40.360 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.355+0000 7efbff7fe640 1 -- 172.21.15.29:0/3992944436 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf00093a0 con 0x7efc001414a0 2023-08-22T23:21:40.361 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.359+0000 7efbff7fe640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 0x7efc001dd9d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c820 tx=0x7efbf400cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:40.361 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.359+0000 7efbfcff9640 10 client.?.objecter ms_handle_connect 0x7efc001414a0 2023-08-22T23:21:40.361 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.359+0000 7efbfcff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:40.370 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4004020 con 0x7efc001414a0 2023-08-22T23:21:40.370 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:40.370 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:40.370 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: dump: 2023-08-22T23:21:40.370 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:40.370 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:40.371 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: _finish_auth 0 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: _check_auth_tickets 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf4004180 con 0x7efc001414a0 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4007830 con 0x7efc001414a0 2023-08-22T23:21:40.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfcff9640 10 monclient: dump: 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:40.373 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 5 monclient: authenticate success, global_id 4160 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 10 monclient: _renew_subs 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:40.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc00139280 con 0x7efc001414a0 2023-08-22T23:21:40.375 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efbfe7fc640 4 set_mon_vals no callback set 2023-08-22T23:21:40.375 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:40.375 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 10 monclient: _renew_subs 2023-08-22T23:21:40.375 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:40.375 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc00139740 con 0x7efc001414a0 2023-08-22T23:21:40.375 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.368+0000 7efc06495640 1 librados: init done 2023-08-22T23:21:40.376 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.370+0000 7efc06495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:40.376 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.370+0000 7efc06495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:40.376 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.370+0000 7efc06495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:40.376 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.370+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0013c4c0 con 0x7efc001414a0 2023-08-22T23:21:40.845 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.843+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 3) v1 ==== 88179+0+0 (secure 0 0 0) 0x7efbf4007990 con 0x7efc001414a0 2023-08-22T23:21:40.845 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.844+0000 7efbfcff9640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7efbec06b880 0x7efbec06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:40.846 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.844+0000 7efbfffff640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7efbec06b880 0x7efbec06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:40.846 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.844+0000 7efbfffff640 10 monclient: get_auth_request con 0x7efbec06b880 auth_method 0 2023-08-22T23:21:40.846 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.845+0000 7efbfcff9640 10 client.4160.objecter ms_handle_connect 0x7efbec06b880 2023-08-22T23:21:40.847 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.845+0000 7efbfffff640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7efbec06b880 0x7efbec06dd40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efbf0036000 tx=0x7efbf0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:40.878 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7efbf405b5c0 con 0x7efc001414a0 2023-08-22T23:21:40.879 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 10 client.4160.objecter ms_dispatch 0x7efc00000b90 osd_map(10..10 src has 1..10) v4 2023-08-22T23:21:40.879 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 3 client.4160.objecter handle_osd_map got epochs [10,10] > 0 2023-08-22T23:21:40.879 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 3 client.4160.objecter handle_osd_map decoding full epoch 10 2023-08-22T23:21:40.879 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 20 client.4160.objecter dump_active .. 0 homeless 2023-08-22T23:21:40.879 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7efbf4053bf0 con 0x7efc001414a0 2023-08-22T23:21:40.879 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:40.880 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:40.877+0000 7efbfcff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:41.123 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.121+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7efbf4053fc0 con 0x7efc001414a0 2023-08-22T23:21:41.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.247+0000 7efc06495640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:41.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.247+0000 7efc06495640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:41.250 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.247+0000 7efc06495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:41.250 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.247+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc0013c4c0 con 0x7efc001414a0 2023-08-22T23:21:41.277 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.275+0000 7efbfcff9640 1 -- 172.21.15.29:0/3992944436 <== mon.2 v2:172.21.15.29:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7efc0013c4c0 con 0x7efc001414a0 2023-08-22T23:21:41.277 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.275+0000 7efbfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:41.277 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.275+0000 7efbfcff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:41.278 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:21:41.278 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:21:38.340112+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:21:41.280 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 10 librados: watch_flush enter 2023-08-22T23:21:41.280 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 10 librados: watch_flush exit 2023-08-22T23:21:41.280 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 20 client.4160.objecter shutdown clearing up homeless session... 2023-08-22T23:21:41.281 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 10 client.4160.objecter successfully canceled tick 2023-08-22T23:21:41.281 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7efbec06b880 msgr2=0x7efbec06dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:41.282 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7efbec06b880 0x7efbec06dd40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efbf0036000 tx=0x7efbf0008030 comp rx=0 tx=0).stop 2023-08-22T23:21:41.282 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 10 monclient: shutdown 2023-08-22T23:21:41.282 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:41.282 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 msgr2=0x7efc001dd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:41.282 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 0x7efc001dd9d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c820 tx=0x7efbf400cce0 comp rx=0 tx=0).stop 2023-08-22T23:21:41.282 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 shutdown_connections 2023-08-22T23:21:41.283 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7efbec06b880 0x7efbec06dd40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:41.283 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc0013a9d0 0x7efc001ddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:41.283 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7efc001414a0 0x7efc001dd9d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:41.283 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 --2- 172.21.15.29:0/3992944436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc0013e800 0x7efc0013ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:41.283 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 >> 172.21.15.29:0/3992944436 conn(0x7efc000b69c0 msgr2=0x7efc000b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:41.283 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 shutdown_connections 2023-08-22T23:21:41.284 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.278+0000 7efc06495640 1 -- 172.21.15.29:0/3992944436 wait complete. 2023-08-22T23:21:41.284 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:41.279+0000 7efc06495640 1 librados: shutdown 2023-08-22T23:21:41.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-22T23:21:46.869 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:08.115 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:46.935 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:08.180 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.010 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:08.248 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.063 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:08.303 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.076 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:08.314 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.124 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:08.364 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.267 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:08.542 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.302 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:08.544 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.329 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:08.603 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.363 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:08.605 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.451 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:08.709 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.490 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:21:47.481+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:47.503 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:08.760 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.511 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:08.770 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.564 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:08.822 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.656 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:21:47.649+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:47.673 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:08.931 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.698 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:21:47.691+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:47.738 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:08.996 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:47.876 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:21:47.868+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:47.942 INFO:tasks.ceph.osd.5.smithi052.stderr:2023-08-22T23:21:47.932+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:47.950 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.196 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.086 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:21:48.078+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:48.100 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:21:48.111 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:21:48.103+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:48.138 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:09.376 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.184 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:09.424 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.213 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.212+0000 7fcbcb18b640 10 monclient: get_monmap_and_config 2023-08-22T23:21:48.214 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.212+0000 7fcbcb18b640 10 monclient: build_initial_monmap 2023-08-22T23:21:48.214 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.212+0000 7fcbcb18b640 10 monclient: monmap: 2023-08-22T23:21:48.214 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:21:48.215 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:48.215 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:21:48.213513+0000 2023-08-22T23:21:48.215 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:21:48.213513+0000 2023-08-22T23:21:48.216 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:48.216 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:48.216 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:48.217 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:48.217 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:48.217 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:48.217 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:48.218 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 Processor -- start 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 -- start start 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient: init 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc40a7bb0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcbc40a7f90 0x7fcbc41419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc40a7f90 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbc413e3b0 0x7fcbc413e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc413e3b0 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40b0d70 con 0x7fcbc413e3b0 2023-08-22T23:21:48.222 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40b0fc0 con 0x7fcbc40a7f90 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc413ed80 con 0x7fcbc40a7bb0 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.213+0000 7fcbcb18b640 10 monclient(hunting): authenticate will time out at 1472.464199s 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc8f27640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcbc40a7f90 0x7fcbc41419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcbc40a7f90 0x7fcbc41419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:41794/0 (socket says 172.21.15.29:41794) 2023-08-22T23:21:48.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 1 -- 172.21.15.29:0/2108606507 learned_addr learned my addr 172.21.15.29:0/2108606507 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:48.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 10 monclient(hunting): get_auth_request con 0x7fcbc40a7f90 auth_method 0 2023-08-22T23:21:48.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:48.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:48.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:48.225 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc8f27640 10 monclient(hunting): get_auth_request con 0x7fcbc40a7bb0 auth_method 0 2023-08-22T23:21:48.225 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc8f27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:48.225 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc8f27640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:48.225 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc8f27640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:48.225 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.214+0000 7fcbc9728640 1 --2- 172.21.15.29:0/2108606507 >> v2:172.21.15.29:3300/0 conn(0x7fcbc413e3b0 0x7fcbc413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.225 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): get_auth_request con 0x7fcbc413e3b0 auth_method 0 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:48.226 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc8f27640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:48.227 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc8f27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:48.227 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc8f27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:48.227 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:48.227 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:48.227 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.215+0000 7fcbc9728640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:48.227 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_done global_id 4195 payload 274 2023-08-22T23:21:48.228 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 1 -- 172.21.15.29:0/2108606507 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 msgr2=0x7fcbc40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:48.228 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 1 --2- 172.21.15.29:0/2108606507 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc40b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.228 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 1 -- 172.21.15.29:0/2108606507 >> v2:172.21.15.29:3300/0 conn(0x7fcbc413e3b0 msgr2=0x7fcbc413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:48.228 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 1 --2- 172.21.15.29:0/2108606507 >> v2:172.21.15.29:3300/0 conn(0x7fcbc413e3b0 0x7fcbc413e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.228 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 10 monclient: _finish_hunting 0 2023-08-22T23:21:48.229 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 1 monclient: found mon.noname-c 2023-08-22T23:21:48.229 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:48.229 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:21:48.229 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.216+0000 7fcbc3fff640 1 -- 172.21.15.29:0/2108606507 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40a9b20 con 0x7fcbc40a7f90 2023-08-22T23:21:48.229 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.218+0000 7fcbc3fff640 1 --2- 172.21.15.29:0/2108606507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41419c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40b3300 tx=0x7fcbb002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00de7d081a87c24 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:48.232 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 1 -- 172.21.15.29:0/2108606507 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0031870 con 0x7fcbc40a7f90 2023-08-22T23:21:48.232 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:48.232 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:48.232 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: dump: 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:48.233 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: _finish_auth 0 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: _check_auth_tickets 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 1 -- 172.21.15.29:0/2108606507 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb00319d0 con 0x7fcbc40a7f90 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbcb18b640 5 monclient: authenticate success, global_id 4195 2023-08-22T23:21:48.234 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbcb18b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbcb18b640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbcb18b640 4 set_mon_vals no callback set 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 1 -- 172.21.15.29:0/2108606507 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0031d60 con 0x7fcbc40a7f90 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.222+0000 7fcbc37fe640 10 monclient: dump: 2023-08-22T23:21:48.235 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:48.236 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:48.236 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.236 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.236 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:48.236 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 10 monclient: shutdown 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 -- 172.21.15.29:0/2108606507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc40a7f90 msgr2=0x7fcbc41419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:48.237 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/2108606507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41419c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40b3300 tx=0x7fcbb002b910 comp rx=0 tx=0).stop 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 -- 172.21.15.29:0/2108606507 shutdown_connections 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/2108606507 >> v2:172.21.15.29:3300/0 conn(0x7fcbc413e3b0 0x7fcbc413e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/2108606507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41419c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/2108606507 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 -- 172.21.15.29:0/2108606507 >> 172.21.15.29:0/2108606507 conn(0x7fcbc40b6920 msgr2=0x7fcbc40b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 -- 172.21.15.29:0/2108606507 shutdown_connections 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.223+0000 7fcbcb18b640 1 -- 172.21.15.29:0/2108606507 wait complete. 2023-08-22T23:21:48.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 10 monclient: build_initial_monmap 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 10 monclient: monmap: 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:48.239 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:48.240 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:48.240 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:48.240 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:48.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 1 librados: starting msgr at 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 1 librados: starting objecter 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 1 Processor -- start 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 1 -- start start 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 1 librados: setting wanted keys 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 1 librados: calling monclient init 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.224+0000 7fcbcb18b640 10 monclient: init 2023-08-22T23:21:48.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:48.242 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:21:48.242 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc4140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.242 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc40a7bb0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:48.242 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.242 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc40a7f90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:48.242 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 0x7fcbc41ddce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.243 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc413e3b0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:48.243 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:48.243 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4143d30 con 0x7fcbc40a7f90 2023-08-22T23:21:48.243 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:48.243 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41441c0 con 0x7fcbc413e3b0 2023-08-22T23:21:48.244 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:48.244 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4140d80 con 0x7fcbc40a7bb0 2023-08-22T23:21:48.244 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:48.244 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbcb18b640 10 monclient(hunting): authenticate will time out at 1472.475798s 2023-08-22T23:21:48.244 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbc9728640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 0x7fcbc41ddce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.225+0000 7fcbc8f27640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc4140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 0x7fcbc41ddce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:41802/0 (socket says 172.21.15.29:41802) 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 1 -- 172.21.15.29:0/3849622232 learned_addr learned my addr 172.21.15.29:0/3849622232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): get_auth_request con 0x7fcbc413e3b0 auth_method 0 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:48.245 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc3fff640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.246 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc8f27640 10 monclient(hunting): get_auth_request con 0x7fcbc40a7bb0 auth_method 0 2023-08-22T23:21:48.246 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc8f27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:48.246 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc8f27640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:48.246 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc8f27640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:48.246 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:48.246 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc9728640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc3fff640 10 monclient(hunting): get_auth_request con 0x7fcbc40a7f90 auth_method 0 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.226+0000 7fcbc3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.227+0000 7fcbc8f27640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:48.247 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.227+0000 7fcbc8f27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.227+0000 7fcbc8f27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.227+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.227+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.227+0000 7fcbc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 10 monclient(hunting): handle_auth_done global_id 4198 payload 824 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 1 -- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 msgr2=0x7fcbc4140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:48.248 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc4140430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 1 -- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc40a7f90 msgr2=0x7fcbc41435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41435d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 10 monclient: _finish_hunting 0 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 1 monclient: found mon.noname-b 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:21:48.249 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.229+0000 7fcbc9728640 1 -- 172.21.15.29:0/3849622232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb00094d0 con 0x7fcbc413e3b0 2023-08-22T23:21:48.250 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.231+0000 7fcbc9728640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 0x7fcbc41ddce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00efc0 tx=0x7fcbbc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:48.250 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.231+0000 7fcbc17fa640 10 client.?.objecter ms_handle_connect 0x7fcbc413e3b0 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.231+0000 7fcbc17fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.234+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc009ba0 con 0x7fcbc413e3b0 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.234+0000 7fcbc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.234+0000 7fcbc17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.234+0000 7fcbc17fa640 10 monclient: dump: 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:48.251 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.252 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.252 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:48.252 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:48.252 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:48.252 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:48.252 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:48.253 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:48.253 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 10 monclient: _finish_auth 0 2023-08-22T23:21:48.253 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 10 monclient: _check_auth_tickets 2023-08-22T23:21:48.253 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:48.254 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc00f040 con 0x7fcbc413e3b0 2023-08-22T23:21:48.254 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 5 monclient: authenticate success, global_id 4198 2023-08-22T23:21:48.254 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 10 monclient: _renew_subs 2023-08-22T23:21:48.254 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:48.255 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc4142ca0 con 0x7fcbc413e3b0 2023-08-22T23:21:48.255 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:48.255 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc0146c0 con 0x7fcbc413e3b0 2023-08-22T23:21:48.255 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:48.255 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:48.255 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc17fa640 10 monclient: dump: 2023-08-22T23:21:48.256 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:48.256 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:48.256 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.256 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:48.257 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:48.257 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:48.257 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:48.257 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:48.257 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:48.257 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbc2ffd640 4 set_mon_vals no callback set 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 10 monclient: _renew_subs 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc41de400 con 0x7fcbc413e3b0 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.235+0000 7fcbcb18b640 1 librados: init done 2023-08-22T23:21:48.258 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.236+0000 7fcbcb18b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:48.259 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.236+0000 7fcbcb18b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:48.259 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.236+0000 7fcbcb18b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:48.259 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.236+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc4074d70 con 0x7fcbc413e3b0 2023-08-22T23:21:48.319 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.564 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.340 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:21:48.330+0000 5dc1300 -1 Falling back to public interface 2023-08-22T23:21:48.361 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:09.635 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.454 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:09.697 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.517 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:09.755 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.535 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.533+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7fcbbc014820 con 0x7fcbc413e3b0 2023-08-22T23:21:48.535 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.534+0000 7fcbc17fa640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcba806b950 0x7fcba806de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:48.536 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.534+0000 7fcbc8f27640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcba806b950 0x7fcba806de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:48.536 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.534+0000 7fcbc8f27640 10 monclient: get_auth_request con 0x7fcba806b950 auth_method 0 2023-08-22T23:21:48.536 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.534+0000 7fcbc17fa640 10 client.4198.objecter ms_handle_connect 0x7fcba806b950 2023-08-22T23:21:48.536 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.534+0000 7fcbc8f27640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcba806b950 0x7fcba806de10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009b40 tx=0x7fcbb8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:48.537 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:09.778 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.547 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fcbbc05c800 con 0x7fcbc413e3b0 2023-08-22T23:21:48.548 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 10 client.4198.objecter ms_dispatch 0x7fcbc4000b90 osd_map(10..10 src has 1..10) v4 2023-08-22T23:21:48.548 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2023-08-22T23:21:48.548 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2023-08-22T23:21:48.548 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 20 client.4198.objecter dump_active .. 0 homeless 2023-08-22T23:21:48.549 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fcbbc054e30 con 0x7fcbc413e3b0 2023-08-22T23:21:48.549 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:48.550 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.546+0000 7fcbc17fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:48.557 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:09.815 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.607 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:09.865 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.641 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.887 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.714 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:09.989 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.729 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.727+0000 7fcbcb18b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:48.729 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.727+0000 7fcbcb18b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:48.729 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.727+0000 7fcbcb18b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:21:48.729 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.727+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbc4143e70 con 0x7fcbc413e3b0 2023-08-22T23:21:48.779 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.777+0000 7fcbc17fa640 1 -- 172.21.15.29:0/3849622232 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fcbbc018020 con 0x7fcbc413e3b0 2023-08-22T23:21:48.779 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.777+0000 7fcbc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:48.779 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.777+0000 7fcbc17fa640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:48.780 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:21:48.780 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:21:38.340112+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:21:48.781 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 10 librados: watch_flush enter 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 10 librados: watch_flush exit 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 20 client.4198.objecter shutdown clearing up homeless session... 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 10 client.4198.objecter successfully canceled tick 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcba806b950 msgr2=0x7fcba806de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcba806b950 0x7fcba806de10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009b40 tx=0x7fcbb8009190 comp rx=0 tx=0).stop 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 10 monclient: shutdown 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:48.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 msgr2=0x7fcbc41ddce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 0x7fcbc41ddce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00efc0 tx=0x7fcbbc00c490 comp rx=0 tx=0).stop 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 shutdown_connections 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcba806b950 0x7fcba806de10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbc413e3b0 0x7fcbc41ddce0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc40a7f90 0x7fcbc41435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 --2- 172.21.15.29:0/3849622232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcbc40a7bb0 0x7fcbc4140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:48.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.779+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 >> 172.21.15.29:0/3849622232 conn(0x7fcbc40b6920 msgr2=0x7fcbc40b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:48.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.780+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 shutdown_connections 2023-08-22T23:21:48.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.780+0000 7fcbcb18b640 1 -- 172.21.15.29:0/3849622232 wait complete. 2023-08-22T23:21:48.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:48.780+0000 7fcbcb18b640 1 librados: shutdown 2023-08-22T23:21:48.790 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-22T23:21:48.811 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.053 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.827 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:10.085 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.833 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.071 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.855 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.095 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.910 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:10.155 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.916 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:10.175 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:48.980 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.235 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.075 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.349 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.113 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.351 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.154 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.394 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.184 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:10.442 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.196 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:10.442 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.197 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.440 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.299 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:10.558 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.342 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.599 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.375 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.649 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.406 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.644 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.447 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.687 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.518 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.761 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.519 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:10.765 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.553 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:10.811 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.595 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:10.854 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.639 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:10.896 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.666 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:10.941 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.695 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.933 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.742 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.982 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.816 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.059 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.829 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:11.075 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.843 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:11.101 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.887 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:11.146 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.930 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:11.188 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.959 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:11.234 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:49.985 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:11.223 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.038 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:11.278 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.109 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.352 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.129 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:11.375 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.135 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:11.393 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.180 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:11.439 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.222 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:11.480 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.255 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:11.530 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.274 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:11.512 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.328 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:11.568 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.398 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.640 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.420 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:11.666 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.425 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:11.683 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.473 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:11.732 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.521 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:11.778 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.545 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:11.819 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.563 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:11.801 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.617 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:11.857 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.688 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.931 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.713 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:11.971 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.768 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:12.027 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.813 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:12.071 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.838 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:12.112 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.948 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:12.194 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:50.983 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:12.226 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.004 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:12.250 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.005 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:12.263 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.058 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:12.317 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.099 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:12.337 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.105 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:12.362 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.156 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:12.394 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.166 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:12.406 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.222 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:12.462 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.281 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:12.539 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.349 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:12.624 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.406 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:12.681 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.515 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:12.757 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.540 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:12.799 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.580 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:12.822 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.609 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:12.868 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.627 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:12.884 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.683 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:12.941 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.822 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:13.080 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:51.892 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:13.150 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:55.597 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:21:55.733 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.731+0000 7f5715257640 10 monclient: get_monmap_and_config 2023-08-22T23:21:55.733 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.731+0000 7f5715257640 10 monclient: build_initial_monmap 2023-08-22T23:21:55.733 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.731+0000 7f5715257640 10 monclient: monmap: 2023-08-22T23:21:55.734 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:21:55.734 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:55.734 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:21:55.732396+0000 2023-08-22T23:21:55.734 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:21:55.732396+0000 2023-08-22T23:21:55.735 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:55.735 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:55.735 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:21:55.736 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:21:55.736 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:21:55.736 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:21:55.736 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:21:55.737 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:55.740 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.732+0000 7f5715257640 1 Processor -- start 2023-08-22T23:21:55.741 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.732+0000 7f5715257640 1 -- start start 2023-08-22T23:21:55.741 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.732+0000 7f5715257640 10 monclient: init 2023-08-22T23:21:55.741 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:55.741 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-08-22T23:21:55.742 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57100a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.742 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient(hunting): picked mon.noname-b con 0x7f571013e800 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:55.742 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f57100a88b0 0x7f57100a8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.743 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient(hunting): picked mon.noname-c con 0x7f57100a88b0 addr v2:172.21.15.52:3300/0 2023-08-22T23:21:55.743 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57101414a0 0x7f5710141930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.743 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient(hunting): picked mon.noname-a con 0x7f57101414a0 addr v2:172.21.15.29:3300/0 2023-08-22T23:21:55.743 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:55.744 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57100a8740 con 0x7f57101414a0 2023-08-22T23:21:55.744 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:55.744 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.733+0000 7f5715257640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57100a8dd0 con 0x7f57100a88b0 2023-08-22T23:21:55.744 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5715257640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:55.745 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5715257640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5710141e70 con 0x7f571013e800 2023-08-22T23:21:55.745 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5715257640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:55.745 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f57100a88b0 0x7f57100a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.745 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5715257640 10 monclient(hunting): authenticate will time out at 1479.984635s 2023-08-22T23:21:55.746 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f570f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57101414a0 0x7f5710141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.746 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f57100a88b0 0x7f57100a8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37266/0 (socket says 172.21.15.29:37266) 2023-08-22T23:21:55.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 1 -- 172.21.15.29:0/1821252387 learned_addr learned my addr 172.21.15.29:0/1821252387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:55.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): get_auth_request con 0x7f57100a88b0 auth_method 0 2023-08-22T23:21:55.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:55.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:55.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:55.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f570ed9d640 1 --2- 172.21.15.29:0/1821252387 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57100a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f570ed9d640 10 monclient(hunting): get_auth_request con 0x7f571013e800 auth_method 0 2023-08-22T23:21:55.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f570ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:55.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f570ed9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:55.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f570ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:55.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:55.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:55.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.734+0000 7f5707fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:55.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): get_auth_request con 0x7f57101414a0 auth_method 0 2023-08-22T23:21:55.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:55.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:55.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:55.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:55.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:55.752 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:55.752 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:55.753 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:55.753 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.735+0000 7f570f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:55.753 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 10 monclient(hunting): handle_auth_done global_id 4201 payload 274 2023-08-22T23:21:55.754 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 1 -- 172.21.15.29:0/1821252387 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 msgr2=0x7f57100a7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:55.754 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 1 --2- 172.21.15.29:0/1821252387 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57100a7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.754 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 1 -- 172.21.15.29:0/1821252387 >> v2:172.21.15.29:3300/0 conn(0x7f57101414a0 msgr2=0x7f5710141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:21:55.754 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 1 --2- 172.21.15.29:0/1821252387 >> v2:172.21.15.29:3300/0 conn(0x7f57101414a0 0x7f5710141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.755 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 10 monclient: _finish_hunting 0 2023-08-22T23:21:55.755 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 1 monclient: found mon.noname-c 2023-08-22T23:21:55.755 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:55.755 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:21:55.756 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.736+0000 7f5707fff640 1 -- 172.21.15.29:0/1821252387 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710141fe0 con 0x7f57100a88b0 2023-08-22T23:21:55.756 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.738+0000 7f5707fff640 1 --2- 172.21.15.29:0/1821252387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57100a88b0 0x7f57100a8200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57100b75a0 tx=0x7f56f802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab0928e520c1561 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:55.759 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 1 -- 172.21.15.29:0/1821252387 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80317e0 con 0x7f57100a88b0 2023-08-22T23:21:55.760 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:55.760 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:55.760 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: dump: 2023-08-22T23:21:55.760 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:55.760 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:55.760 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:55.761 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:55.762 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: _finish_auth 0 2023-08-22T23:21:55.762 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: _check_auth_tickets 2023-08-22T23:21:55.762 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:55.762 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 1 -- 172.21.15.29:0/1821252387 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8031940 con 0x7f57100a88b0 2023-08-22T23:21:55.762 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:55.762 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 1 -- 172.21.15.29:0/1821252387 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8031cd0 con 0x7f57100a88b0 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f570e59c640 10 monclient: dump: 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.763 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:55.764 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f5715257640 5 monclient: authenticate success, global_id 4201 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f5715257640 10 monclient: get_monmap_and_config success 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.743+0000 7f5715257640 4 set_mon_vals no callback set 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 10 monclient: shutdown 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 -- 172.21.15.29:0/1821252387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57100a88b0 msgr2=0x7f57100a8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 --2- 172.21.15.29:0/1821252387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57100a88b0 0x7f57100a8200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57100b75a0 tx=0x7f56f802b880 comp rx=0 tx=0).stop 2023-08-22T23:21:55.765 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 -- 172.21.15.29:0/1821252387 shutdown_connections 2023-08-22T23:21:55.766 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 --2- 172.21.15.29:0/1821252387 >> v2:172.21.15.29:3300/0 conn(0x7f57101414a0 0x7f5710141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.766 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 --2- 172.21.15.29:0/1821252387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57100a88b0 0x7f57100a8200 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.766 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 --2- 172.21.15.29:0/1821252387 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57100a7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.766 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 -- 172.21.15.29:0/1821252387 >> 172.21.15.29:0/1821252387 conn(0x7f57100b69c0 msgr2=0x7f57100b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:55.766 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 -- 172.21.15.29:0/1821252387 shutdown_connections 2023-08-22T23:21:55.766 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.744+0000 7f5715257640 1 -- 172.21.15.29:0/1821252387 wait complete. 2023-08-22T23:21:55.767 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.745+0000 7f5715257640 10 monclient: build_initial_monmap 2023-08-22T23:21:55.767 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.745+0000 7f5715257640 10 monclient: monmap: 2023-08-22T23:21:55.767 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:21:55.767 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:21:55.767 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:21:55.767 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:21:55.768 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:21:55.768 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:55.768 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:21:55.768 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:21:55.768 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:21:55.768 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:55.769 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.745+0000 7f5715257640 1 librados: starting msgr at 2023-08-22T23:21:55.769 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.745+0000 7f5715257640 1 librados: starting objecter 2023-08-22T23:21:55.769 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 1 Processor -- start 2023-08-22T23:21:55.769 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 1 -- start start 2023-08-22T23:21:55.770 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 1 librados: setting wanted keys 2023-08-22T23:21:55.770 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 1 librados: calling monclient init 2023-08-22T23:21:55.770 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 10 monclient: init 2023-08-22T23:21:55.770 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 10 monclient: _reopen_session rank -1 2023-08-22T23:21:55.770 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:21:55.770 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.746+0000 7f5715257640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57101d60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.771 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): picked mon.noname-c con 0x7f571013e800 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:21:55.771 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57101414a0 0x7f57100ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.771 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): picked mon.noname-b con 0x7f57101414a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:21:55.771 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5710144120 0x7f57101d6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.771 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): picked mon.noname-a con 0x7f5710144120 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:21:55.771 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5710142830 con 0x7f5710144120 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f570ed9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57101d60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f570f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5710144120 0x7f57101d6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5707fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57101414a0 0x7f57100ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57100accd0 con 0x7f57101414a0 2023-08-22T23:21:55.772 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): start opening mon connection 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57100ac0c0 con 0x7f571013e800 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): _renew_subs 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.747+0000 7f5715257640 10 monclient(hunting): authenticate will time out at 1479.998359s 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57101d60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:35140/0 (socket says 172.21.15.29:35140) 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57101414a0 0x7f57100ac790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37282/0 (socket says 172.21.15.29:37282) 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 1 -- 172.21.15.29:0/2456670647 learned_addr learned my addr 172.21.15.29:0/2456670647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:21:55.773 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): get_auth_request con 0x7f571013e800 auth_method 0 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): get_auth_request con 0x7f57101414a0 auth_method 0 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:55.774 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): get_auth_request con 0x7f5710144120 auth_method 0 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:55.775 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:55.776 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:55.776 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:55.776 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f5707fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:55.776 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:21:55.776 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:21:55.776 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.748+0000 7f570ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 10 monclient(hunting): handle_auth_done global_id 4181 payload 824 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 1 -- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57101414a0 msgr2=0x7f57100ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57101414a0 0x7f57100ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 1 -- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5710144120 msgr2=0x7f57101d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5710144120 0x7f57101d6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 10 monclient: _finish_hunting 0 2023-08-22T23:21:55.777 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 1 monclient: found mon.noname-c 2023-08-22T23:21:55.778 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:21:55.778 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:21:55.778 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.752+0000 7f570ed9d640 1 -- 172.21.15.29:0/2456670647 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f80094d0 con 0x7f571013e800 2023-08-22T23:21:55.778 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.756+0000 7f570ed9d640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57101d60f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f570000e930 tx=0x7f570000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:55.778 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.756+0000 7f57077fe640 10 client.?.objecter ms_handle_connect 0x7f571013e800 2023-08-22T23:21:55.778 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.756+0000 7f57077fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:21:55.780 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700009660 con 0x7f571013e800 2023-08-22T23:21:55.780 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:55.780 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:21:55.780 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 10 monclient: dump: 2023-08-22T23:21:55.780 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:55.780 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:55.781 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 10 monclient: _finish_auth 0 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 10 monclient: _check_auth_tickets 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570000ca00 con 0x7f571013e800 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 5 monclient: authenticate success, global_id 4181 2023-08-22T23:21:55.782 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 10 monclient: _renew_subs 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5710142450 con 0x7f571013e800 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 10 monclient: _renew_subs 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5710142bc0 con 0x7f571013e800 2023-08-22T23:21:55.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.761+0000 7f5715257640 1 librados: init done 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.762+0000 7f57077fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.762+0000 7f570dd9b640 4 set_mon_vals no callback set 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.762+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570000cd70 con 0x7f571013e800 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.762+0000 7f57077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.763+0000 7f57077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.763+0000 7f57077fe640 10 monclient: dump: 2023-08-22T23:21:55.784 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:21:55.785 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:21:55.786 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:21:55.786 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:21:55.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.763+0000 7f5715257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:55.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.763+0000 7f5715257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:55.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.763+0000 7f5715257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:55.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.763+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5710143b60 con 0x7f571013e800 2023-08-22T23:21:55.797 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.796+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7f5700026480 con 0x7f571013e800 2023-08-22T23:21:55.798 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.796+0000 7f57077fe640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56f006b9d0 0x7f56f006de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:21:55.798 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.797+0000 7f5707fff640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56f006b9d0 0x7f56f006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:21:55.799 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.797+0000 7f5707fff640 10 monclient: get_auth_request con 0x7f56f006b9d0 auth_method 0 2023-08-22T23:21:55.799 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.797+0000 7f57077fe640 10 client.4181.objecter ms_handle_connect 0x7f56f006b9d0 2023-08-22T23:21:55.799 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.797+0000 7f5707fff640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56f006b9d0 0x7f56f006de90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f56f8036000 tx=0x7f56f803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:21:55.830 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.828+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f570005bf30 con 0x7f571013e800 2023-08-22T23:21:55.831 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.828+0000 7f57077fe640 10 client.4181.objecter ms_dispatch 0x7f5710000b90 osd_map(10..10 src has 1..10) v4 2023-08-22T23:21:55.831 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.829+0000 7f57077fe640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2023-08-22T23:21:55.831 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.829+0000 7f57077fe640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2023-08-22T23:21:55.831 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:55.829+0000 7f57077fe640 20 client.4181.objecter dump_active .. 0 homeless 2023-08-22T23:21:56.111 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.109+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f5700054560 con 0x7f571013e800 2023-08-22T23:21:56.111 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.109+0000 7f57077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:21:56.111 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.109+0000 7f57077fe640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:21:56.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.390+0000 7f5715257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:56.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.390+0000 7f5715257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:56.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.390+0000 7f5715257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:21:56.393 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.390+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5710143b60 con 0x7f571013e800 2023-08-22T23:21:56.421 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.419+0000 7f57077fe640 1 -- 172.21.15.29:0/2456670647 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5710143b60 con 0x7f571013e800 2023-08-22T23:21:56.421 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.419+0000 7f57077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:21:56.422 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.419+0000 7f57077fe640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:21:56.423 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:21:56.423 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:21:38.340112+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:21:56.425 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 10 librados: watch_flush enter 2023-08-22T23:21:56.426 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 10 librados: watch_flush exit 2023-08-22T23:21:56.426 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 20 client.4181.objecter shutdown clearing up homeless session... 2023-08-22T23:21:56.426 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 10 client.4181.objecter successfully canceled tick 2023-08-22T23:21:56.427 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56f006b9d0 msgr2=0x7f56f006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:56.427 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56f006b9d0 0x7f56f006de90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f56f8036000 tx=0x7f56f803d040 comp rx=0 tx=0).stop 2023-08-22T23:21:56.428 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 10 monclient: shutdown 2023-08-22T23:21:56.428 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:21:56.428 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 msgr2=0x7f57101d60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:21:56.429 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.421+0000 7f5715257640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57101d60f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f570000e930 tx=0x7f570000edf0 comp rx=0 tx=0).stop 2023-08-22T23:21:56.429 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 shutdown_connections 2023-08-22T23:21:56.429 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56f006b9d0 0x7f56f006de90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:56.430 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5710144120 0x7f57101d6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:56.430 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57101414a0 0x7f57100ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:56.430 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 --2- 172.21.15.29:0/2456670647 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f571013e800 0x7f57101d60f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:21:56.430 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 >> 172.21.15.29:0/2456670647 conn(0x7f57100b69c0 msgr2=0x7f57100a7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:21:56.431 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 shutdown_connections 2023-08-22T23:21:56.431 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 -- 172.21.15.29:0/2456670647 wait complete. 2023-08-22T23:21:56.431 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:21:56.422+0000 7f5715257640 1 librados: shutdown 2023-08-22T23:21:56.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-22T23:21:56.649 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:17.887 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:56.817 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:18.063 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:56.859 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:18.099 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:57.212 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:18.486 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:57.219 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:18.478 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:57.445 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:18.702 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:57.504 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:18.746 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:21:57.570 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:18.828 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.183 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:23.422 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.188 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:23.426 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.701 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:23.947 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.706 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:23.951 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.841 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:24.081 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.845 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:24.086 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.892 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:24.135 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:02.896 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:24.139 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.061 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:24.335 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.065 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:24.340 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.158 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:24.415 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.162 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:24.420 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.242 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:22:03.253 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:24.512 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.258 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:24.517 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.348 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.347+0000 7f7667428640 10 monclient: get_monmap_and_config 2023-08-22T23:22:03.349 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.347+0000 7f7667428640 10 monclient: build_initial_monmap 2023-08-22T23:22:03.349 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.347+0000 7f7667428640 10 monclient: monmap: 2023-08-22T23:22:03.349 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:03.349 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:03.349 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:22:03.348616+0000 2023-08-22T23:22:03.349 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:22:03.348616+0000 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:03.350 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:03.351 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:03.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.347+0000 7f7667428640 1 Processor -- start 2023-08-22T23:22:03.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 1 -- start start 2023-08-22T23:22:03.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient: init 2023-08-22T23:22:03.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:03.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-08-22T23:22:03.352 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f766013e800 0x7f76600a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient(hunting): picked mon.noname-c con 0x7f766013e800 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76600a88b0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7660141bb0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600a8dd0 con 0x7f76600a88b0 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7660141fe0 con 0x7f7660141bb0 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766013ebe0 con 0x7f766013e800 2023-08-22T23:22:03.353 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.348+0000 7f7667428640 10 monclient(hunting): authenticate will time out at 1487.599352s 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.349+0000 7f76651c4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f766013e800 0x7f76600a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.350+0000 7f76659c5640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7660141bb0 0x7f766014b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:33458/0 (socket says 172.21.15.29:33458) 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.350+0000 7f76659c5640 1 -- 172.21.15.29:0/2804005606 learned_addr learned my addr 172.21.15.29:0/2804005606 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.350+0000 7f76651c4640 10 monclient(hunting): get_auth_request con 0x7f766013e800 auth_method 0 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.350+0000 7f76651c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:03.354 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.350+0000 7f76651c4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:03.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.350+0000 7f76651c4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:03.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.351+0000 7f76651c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:03.355 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.351+0000 7f76651c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.351+0000 7f76651c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 10 monclient(hunting): handle_auth_done global_id 4207 payload 274 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 1 -- 172.21.15.29:0/2804005606 >> v1:172.21.15.52:6789/0 conn(0x7f7660141bb0 legacy=0x7f766014b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 1 -- 172.21.15.29:0/2804005606 >> v1:172.21.15.29:6789/0 conn(0x7f76600a88b0 legacy=0x7f76601414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 10 monclient: _finish_hunting 0 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 1 monclient: found mon.noname-c 2023-08-22T23:22:03.356 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:03.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:03.357 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.354+0000 7f76651c4640 1 -- 172.21.15.29:0/2804005606 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766013ed20 con 0x7f766013e800 2023-08-22T23:22:03.361 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.360+0000 7f76651c4640 1 --2- 172.21.15.29:0/2804005606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600a7cc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f765c009790 tx=0x7f765c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa5eb165d42a4c70 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:03.371 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 1 -- 172.21.15.29:0/2804005606 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c02e070 con 0x7f766013e800 2023-08-22T23:22:03.371 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:03.371 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: dump: 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:03.372 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: _finish_auth 0 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: _check_auth_tickets 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:03.373 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 1 -- 172.21.15.29:0/2804005606 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c0315a0 con 0x7f766013e800 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 5 monclient: authenticate success, global_id 4207 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 1 -- 172.21.15.29:0/2804005606 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c031930 con 0x7f766013e800 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 4 set_mon_vals no callback set 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:03.374 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7657fff640 10 monclient: dump: 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:03.375 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:03.376 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:03.376 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 10 monclient: shutdown 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 1 -- 172.21.15.29:0/2804005606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 msgr2=0x7f76600a7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 1 --2- 172.21.15.29:0/2804005606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600a7cc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f765c009790 tx=0x7f765c02b840 comp rx=0 tx=0).stop 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 1 -- 172.21.15.29:0/2804005606 shutdown_connections 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 1 --2- 172.21.15.29:0/2804005606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600a7cc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:03.378 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.370+0000 7f7667428640 1 -- 172.21.15.29:0/2804005606 >> 172.21.15.29:0/2804005606 conn(0x7f76600b69c0 msgr2=0x7f76600b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.371+0000 7f7667428640 1 -- 172.21.15.29:0/2804005606 shutdown_connections 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.371+0000 7f7667428640 1 -- 172.21.15.29:0/2804005606 wait complete. 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.371+0000 7f7667428640 10 monclient: build_initial_monmap 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.371+0000 7f7667428640 10 monclient: monmap: 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:03.379 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:03.380 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.371+0000 7f7667428640 1 librados: starting msgr at 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.371+0000 7f7667428640 1 librados: starting objecter 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 Processor -- start 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 -- start start 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 librados: setting wanted keys 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 librados: calling monclient init 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient: init 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:03.381 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): picked mon.noname-b con 0x7f766013e800 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7660141bb0 0x7f76601d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): picked mon.noname-a con 0x7f7660141bb0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f76601d9390 0x7f76601d9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): picked mon.noname-c con 0x7f76601d9390 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:03.382 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7660121ff0 con 0x7f7660141bb0 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76601d6740 con 0x7f766013e800 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76601d9d40 con 0x7f76601d9390 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.372+0000 7f7667428640 10 monclient(hunting): authenticate will time out at 1487.623108s 2023-08-22T23:22:03.383 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76659c5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f76601d9390 0x7f76601d9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600ac540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:45656/0 (socket says 172.21.15.29:45656) 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 1 -- 172.21.15.29:0/2343964302 learned_addr learned my addr 172.21.15.29:0/2343964302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 10 monclient(hunting): get_auth_request con 0x7f766013e800 auth_method 0 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76651c4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:03.384 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76659c5640 10 monclient(hunting): get_auth_request con 0x7f76601d9390 auth_method 0 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76659c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76659c5640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76659c5640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76649c3640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7660141bb0 0x7f76601d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76649c3640 10 monclient(hunting): get_auth_request con 0x7f7660141bb0 auth_method 0 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76649c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:03.385 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76649c3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.373+0000 7f76649c3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76651c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76651c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76651c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76659c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76659c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76659c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:03.386 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76649c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:03.387 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76649c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:03.387 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.374+0000 7f76649c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:03.387 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.377+0000 7f76649c3640 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2023-08-22T23:22:03.387 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 1 -- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f76601d9390 msgr2=0x7f76601d9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f76601d9390 0x7f76601d9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 1 -- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 msgr2=0x7f76600ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600ac540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 10 monclient: _finish_hunting 0 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 1 monclient: found mon.noname-a 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:03.388 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.378+0000 7f76649c3640 1 -- 172.21.15.29:0/2343964302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c0093a0 con 0x7f7660141bb0 2023-08-22T23:22:03.389 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:24.640 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.389 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.382+0000 7f76649c3640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7660141bb0 0x7f76601d5e10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f765000c440 tx=0x7f765000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.382+0000 7f7655ffb640 10 client.?.objecter ms_handle_connect 0x7f7660141bb0 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.382+0000 7f7655ffb640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76500035f0 con 0x7f7660141bb0 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: dump: 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:03.390 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:03.391 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: _finish_auth 0 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7650003770 con 0x7f7660141bb0 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650003b20 con 0x7f7660141bb0 2023-08-22T23:22:03.392 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:03.393 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:24.647 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:03.393 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7655ffb640 10 monclient: dump: 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:03.394 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7667428640 5 monclient: authenticate success, global_id 4215 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7667428640 10 monclient: _renew_subs 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7667428640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76601d6350 con 0x7f7660141bb0 2023-08-22T23:22:03.395 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.387+0000 7f76577fe640 4 set_mon_vals no callback set 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.388+0000 7f7667428640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.388+0000 7f7667428640 10 monclient: _renew_subs 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.388+0000 7f7667428640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.388+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76601d6bd0 con 0x7f7660141bb0 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.388+0000 7f7667428640 1 librados: init done 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.389+0000 7f7667428640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:03.396 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.389+0000 7f7667428640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:03.397 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.389+0000 7f7667428640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:03.397 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.389+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76601d64e0 con 0x7f7660141bb0 2023-08-22T23:22:03.725 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.723+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7f7650003c80 con 0x7f7660141bb0 2023-08-22T23:22:03.726 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.724+0000 7f7655ffb640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f764406b9a0 0x7f764406de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:03.726 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.724+0000 7f76651c4640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f764406b9a0 0x7f764406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:03.726 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.724+0000 7f76651c4640 10 monclient: get_auth_request con 0x7f764406b9a0 auth_method 0 2023-08-22T23:22:03.726 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.724+0000 7f76651c4640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f764406b9a0 0x7f764406de60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f765c02b880 tx=0x7f765c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:03.726 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.724+0000 7f7655ffb640 10 client.4215.objecter ms_handle_connect 0x7f764406b9a0 2023-08-22T23:22:03.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f765005b110 con 0x7f7660141bb0 2023-08-22T23:22:03.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 10 client.4215.objecter ms_dispatch 0x7f7660000b90 osd_map(10..10 src has 1..10) v4 2023-08-22T23:22:03.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2023-08-22T23:22:03.747 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2023-08-22T23:22:03.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 20 client.4215.objecter dump_active .. 0 homeless 2023-08-22T23:22:03.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f7650053740 con 0x7f7660141bb0 2023-08-22T23:22:03.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:03.748 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:03.745+0000 7f7655ffb640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:04.006 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.004+0000 7f7667428640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:04.006 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.004+0000 7f7667428640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:04.006 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.004+0000 7f7667428640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:04.006 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.004+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7660066cb0 con 0x7f7660141bb0 2023-08-22T23:22:04.094 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.093+0000 7f7655ffb640 1 -- 172.21.15.29:0/2343964302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7650017070 con 0x7f7660141bb0 2023-08-22T23:22:04.095 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.093+0000 7f7655ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:04.095 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.093+0000 7f7655ffb640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:04.095 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:22:04.095 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:21:38.340112+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:22:04.097 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.095+0000 7f7667428640 10 librados: watch_flush enter 2023-08-22T23:22:04.098 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.095+0000 7f7667428640 10 librados: watch_flush exit 2023-08-22T23:22:04.098 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.095+0000 7f7667428640 20 client.4215.objecter shutdown clearing up homeless session... 2023-08-22T23:22:04.098 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.095+0000 7f7667428640 10 client.4215.objecter successfully canceled tick 2023-08-22T23:22:04.098 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.095+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f764406b9a0 msgr2=0x7f764406de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:04.098 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.095+0000 7f7667428640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f764406b9a0 0x7f764406de60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f765c02b880 tx=0x7f765c03d040 comp rx=0 tx=0).stop 2023-08-22T23:22:04.098 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 10 monclient: shutdown 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7660141bb0 msgr2=0x7f76601d5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7660141bb0 0x7f76601d5e10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f765000c440 tx=0x7f765000c840 comp rx=0 tx=0).stop 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 shutdown_connections 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f764406b9a0 0x7f764406de60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f76601d9390 0x7f76601d9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:04.099 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7660141bb0 0x7f76601d5e10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:04.100 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 --2- 172.21.15.29:0/2343964302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f766013e800 0x7f76600ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:04.100 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 >> 172.21.15.29:0/2343964302 conn(0x7f76600b69c0 msgr2=0x7f76600b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:04.100 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 shutdown_connections 2023-08-22T23:22:04.100 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 -- 172.21.15.29:0/2343964302 wait complete. 2023-08-22T23:22:04.100 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:04.096+0000 7f7667428640 1 librados: shutdown 2023-08-22T23:22:04.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-22T23:22:06.382 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:27.620 86898== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.008 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:28.253 86895== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.219 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:28.461 86896== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.262 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:28.502 86897== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.682 INFO:tasks.ceph.osd.2.smithi029.stderr:==00:00:00:28.956 87043== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.780 INFO:tasks.ceph.osd.3.smithi029.stderr:==00:00:00:29.037 87048== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.811 INFO:tasks.ceph.osd.1.smithi029.stderr:==00:00:00:29.069 87049== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:07.970 INFO:tasks.ceph.osd.0.smithi029.stderr:==00:00:00:29.228 87046== Warning: unimplemented fcntl command: 1038 2023-08-22T23:22:10.916 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:22:11.049 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.047+0000 7fb09a2aa640 10 monclient: get_monmap_and_config 2023-08-22T23:22:11.049 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.047+0000 7fb09a2aa640 10 monclient: build_initial_monmap 2023-08-22T23:22:11.049 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.047+0000 7fb09a2aa640 10 monclient: monmap: 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:22:11.048693+0000 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:22:11.048693+0000 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:11.050 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:11.051 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:11.051 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:11.051 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:11.051 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:11.052 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 1 Processor -- start 2023-08-22T23:22:11.052 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 1 -- start start 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 10 monclient: init 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb09413e800 0x7fb0940a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.048+0000 7fb09a2aa640 10 monclient(hunting): picked mon.noname-b con 0x7fb09413e800 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0940a88b0 0x7fb0940a8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:11.053 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 10 monclient(hunting): picked mon.noname-a con 0x7fb0940a88b0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0941414a0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940a8dd0 con 0x7fb0941414a0 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0940a8740 con 0x7fb0940a88b0 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb094142040 con 0x7fb09413e800 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:11.054 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb09a2aa640 10 monclient(hunting): authenticate will time out at 1495.299671s 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb0937fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0940a88b0 0x7fb0940a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb098847640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb0941414a0 0x7fb094141930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:39772/0 (socket says 172.21.15.29:39772) 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.049+0000 7fb098847640 1 -- 172.21.15.29:0/817097887 learned_addr learned my addr 172.21.15.29:0/817097887 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.050+0000 7fb0937fe640 10 monclient(hunting): get_auth_request con 0x7fb0940a88b0 auth_method 0 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.050+0000 7fb0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.050+0000 7fb0937fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:11.055 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.050+0000 7fb0937fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:11.056 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.051+0000 7fb0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:11.056 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.051+0000 7fb0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:11.056 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.051+0000 7fb0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:11.056 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2023-08-22T23:22:11.056 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 1 -- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb09413e800 msgr2=0x7fb0940a7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 1 --2- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb09413e800 0x7fb0940a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 1 -- 172.21.15.29:0/817097887 >> v1:172.21.15.52:6789/0 conn(0x7fb0941414a0 legacy=0x7fb094141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 1 monclient: found mon.noname-a 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:11.057 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.053+0000 7fb0937fe640 1 -- 172.21.15.29:0/817097887 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09413ebe0 con 0x7fb0940a88b0 2023-08-22T23:22:11.058 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.056+0000 7fb0937fe640 1 --2- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0940a88b0 0x7fb0940a8200 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0880097d0 tx=0x7fb08802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227b133372674c68 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:11.063 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 1 -- 172.21.15.29:0/817097887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088031780 con 0x7fb0940a88b0 2023-08-22T23:22:11.063 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:11.064 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:11.064 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: dump: 2023-08-22T23:22:11.064 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:11.064 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:11.064 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.064 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:11.065 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 1 -- 172.21.15.29:0/817097887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0880318e0 con 0x7fb0940a88b0 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb09a2aa640 5 monclient: authenticate success, global_id 4218 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb09a2aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb09a2aa640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 1 -- 172.21.15.29:0/817097887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088031c70 con 0x7fb0940a88b0 2023-08-22T23:22:11.066 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb09a2aa640 4 set_mon_vals no callback set 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.062+0000 7fb092ffd640 10 monclient: dump: 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.067 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.068 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:11.068 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:11.068 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:11.068 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:11.068 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:11.068 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:11.070 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 10 monclient: shutdown 2023-08-22T23:22:11.071 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:11.071 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 -- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0940a88b0 msgr2=0x7fb0940a8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:11.071 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0940a88b0 0x7fb0940a8200 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0880097d0 tx=0x7fb08802b820 comp rx=0 tx=0).stop 2023-08-22T23:22:11.071 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 -- 172.21.15.29:0/817097887 shutdown_connections 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0940a88b0 0x7fb0940a8200 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/817097887 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb09413e800 0x7fb0940a7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 -- 172.21.15.29:0/817097887 >> 172.21.15.29:0/817097887 conn(0x7fb0940b69c0 msgr2=0x7fb0940b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 -- 172.21.15.29:0/817097887 shutdown_connections 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.063+0000 7fb09a2aa640 1 -- 172.21.15.29:0/817097887 wait complete. 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.064+0000 7fb09a2aa640 10 monclient: build_initial_monmap 2023-08-22T23:22:11.072 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.065+0000 7fb09a2aa640 10 monclient: monmap: 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:11.073 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.065+0000 7fb09a2aa640 1 librados: starting msgr at 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.065+0000 7fb09a2aa640 1 librados: starting objecter 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.065+0000 7fb09a2aa640 1 Processor -- start 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 1 -- start start 2023-08-22T23:22:11.074 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 1 librados: setting wanted keys 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 1 librados: calling monclient init 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 10 monclient: init 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.066+0000 7fb09a2aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 0x7fb0941426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): picked mon.noname-a con 0x7fb09413e800 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb0941414a0 0x7fb09413d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:11.075 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): picked mon.noname-c con 0x7fb0941414a0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:11.076 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0941d96d0 0x7fb0941449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:11.076 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): picked mon.noname-b con 0x7fb0941d96d0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:11.076 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:11.076 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941431a0 con 0x7fb09413e800 2023-08-22T23:22:11.076 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094144220 con 0x7fb0941d96d0 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941444c0 con 0x7fb0941414a0 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb09a2aa640 10 monclient(hunting): authenticate will time out at 1495.317660s 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb093fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 0x7fb0941426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:11.077 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.067+0000 7fb098847640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0941d96d0 0x7fb0941449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 0x7fb0941426c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51816/0 (socket says 172.21.15.29:51816) 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 1 -- 172.21.15.29:0/1321695948 learned_addr learned my addr 172.21.15.29:0/1321695948 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): get_auth_request con 0x7fb09413e800 auth_method 0 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb098847640 10 monclient(hunting): get_auth_request con 0x7fb0941d96d0 auth_method 0 2023-08-22T23:22:11.078 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb098847640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb098847640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb098847640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.068+0000 7fb093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.069+0000 7fb098847640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.069+0000 7fb098847640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:11.079 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.069+0000 7fb098847640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:11.080 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 10 monclient(hunting): handle_auth_done global_id 4221 payload 824 2023-08-22T23:22:11.080 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 1 -- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb0941414a0 msgr2=0x7fb09413d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb0941414a0 0x7fb09413d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 1 -- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0941d96d0 msgr2=0x7fb0941449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0941d96d0 0x7fb0941449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 1 monclient: found mon.noname-a 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:11.081 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.073+0000 7fb093fff640 1 -- 172.21.15.29:0/1321695948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088009400 con 0x7fb09413e800 2023-08-22T23:22:11.082 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.077+0000 7fb093fff640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 0x7fb0941426c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb08400b4a0 tx=0x7fb08400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:11.082 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.077+0000 7fb090ff9640 10 client.?.objecter ms_handle_connect 0x7fb09413e800 2023-08-22T23:22:11.082 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.077+0000 7fb090ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:11.085 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084005a10 con 0x7fb09413e800 2023-08-22T23:22:11.085 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: dump: 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:11.086 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:11.087 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084005b70 con 0x7fb09413e800 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084010610 con 0x7fb09413e800 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb090ff9640 10 monclient: dump: 2023-08-22T23:22:11.088 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:11.089 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:11.090 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:11.090 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 5 monclient: authenticate success, global_id 4221 2023-08-22T23:22:11.090 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 10 monclient: _renew_subs 2023-08-22T23:22:11.090 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb0927fc640 4 set_mon_vals no callback set 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094142c00 con 0x7fb09413e800 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 10 monclient: _renew_subs 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094144f20 con 0x7fb09413e800 2023-08-22T23:22:11.091 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.083+0000 7fb09a2aa640 1 librados: init done 2023-08-22T23:22:11.092 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.084+0000 7fb09a2aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:11.092 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.085+0000 7fb09a2aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:11.092 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.085+0000 7fb09a2aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:11.093 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.085+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094066cb0 con 0x7fb09413e800 2023-08-22T23:22:11.122 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.121+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7fb084010770 con 0x7fb09413e800 2023-08-22T23:22:11.123 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.121+0000 7fb090ff9640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb08006b950 0x7fb08006de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:11.123 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.121+0000 7fb0937fe640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb08006b950 0x7fb08006de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:11.123 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.121+0000 7fb0937fe640 10 monclient: get_auth_request con 0x7fb08006b950 auth_method 0 2023-08-22T23:22:11.124 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.122+0000 7fb090ff9640 10 client.4221.objecter ms_handle_connect 0x7fb08006b950 2023-08-22T23:22:11.124 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.122+0000 7fb0937fe640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb08006b950 0x7fb08006de10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb088036000 tx=0x7fb088030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:11.493 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.492+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb08405b380 con 0x7fb09413e800 2023-08-22T23:22:11.493 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.492+0000 7fb090ff9640 10 client.4221.objecter ms_dispatch 0x7fb094000b90 osd_map(10..10 src has 1..10) v4 2023-08-22T23:22:11.494 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.492+0000 7fb090ff9640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2023-08-22T23:22:11.494 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.492+0000 7fb090ff9640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2023-08-22T23:22:11.494 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.492+0000 7fb090ff9640 20 client.4221.objecter dump_active .. 0 homeless 2023-08-22T23:22:11.527 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.526+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fb084057440 con 0x7fb09413e800 2023-08-22T23:22:11.528 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.526+0000 7fb090ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:11.528 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.526+0000 7fb090ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:11.730 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.728+0000 7fb09a2aa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:11.730 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.728+0000 7fb09a2aa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:11.730 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.728+0000 7fb09a2aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:11.731 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.728+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb094141910 con 0x7fb09413e800 2023-08-22T23:22:11.746 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.744+0000 7fb090ff9640 1 -- 172.21.15.29:0/1321695948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb084014020 con 0x7fb09413e800 2023-08-22T23:22:11.746 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.744+0000 7fb090ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:11.746 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.744+0000 7fb090ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:11.747 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:22:11.747 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:21:38.340112+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:22:11.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 10 librados: watch_flush enter 2023-08-22T23:22:11.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 10 librados: watch_flush exit 2023-08-22T23:22:11.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 20 client.4221.objecter shutdown clearing up homeless session... 2023-08-22T23:22:11.749 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 10 client.4221.objecter successfully canceled tick 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb08006b950 msgr2=0x7fb08006de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb08006b950 0x7fb08006de10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb088036000 tx=0x7fb088030000 comp rx=0 tx=0).stop 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 10 monclient: shutdown 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 msgr2=0x7fb0941426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 0x7fb0941426c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb08400b4a0 tx=0x7fb08400b960 comp rx=0 tx=0).stop 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 shutdown_connections 2023-08-22T23:22:11.750 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb08006b950 0x7fb08006de10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0941d96d0 0x7fb0941449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb0941414a0 0x7fb09413d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 --2- 172.21.15.29:0/1321695948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09413e800 0x7fb0941426c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 >> 172.21.15.29:0/1321695948 conn(0x7fb0940b69c0 msgr2=0x7fb09413e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 shutdown_connections 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 -- 172.21.15.29:0/1321695948 wait complete. 2023-08-22T23:22:11.751 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:11.747+0000 7fb09a2aa640 1 librados: shutdown 2023-08-22T23:22:11.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-22T23:22:15.041 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:22:15.036+0000 5dc1300 -1 osd.7 0 log_to_monitors true 2023-08-22T23:22:15.580 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:22:15.576+0000 5dc1300 -1 osd.6 0 log_to_monitors true 2023-08-22T23:22:15.670 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:22:15.666+0000 5dc1300 -1 osd.4 0 log_to_monitors true 2023-08-22T23:22:16.020 INFO:tasks.ceph.osd.5.smithi052.stderr:2023-08-22T23:22:16.015+0000 5dc1300 -1 osd.5 0 log_to_monitors true 2023-08-22T23:22:16.290 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:22:16.280+0000 5dc1300 -1 osd.3 0 log_to_monitors true 2023-08-22T23:22:16.570 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:22:16.556+0000 5dc1300 -1 osd.1 0 log_to_monitors true 2023-08-22T23:22:16.593 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:22:16.590+0000 5dc1300 -1 osd.2 0 log_to_monitors true 2023-08-22T23:22:16.919 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:22:16.916+0000 5dc1300 -1 osd.0 0 log_to_monitors true 2023-08-22T23:22:18.214 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:22:18.209+0000 ac7c640 -1 osd.7 0 waiting for initial osdmap 2023-08-22T23:22:18.565 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:22:18.591 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:22:18.576+0000 13a60640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:18.657 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.657+0000 7f0ad48a9640 10 monclient: get_monmap_and_config 2023-08-22T23:22:18.658 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.657+0000 7f0ad48a9640 10 monclient: build_initial_monmap 2023-08-22T23:22:18.658 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.657+0000 7f0ad48a9640 10 monclient: monmap: 2023-08-22T23:22:18.658 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:18.658 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:18.658 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:22:18.657571+0000 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:22:18.657571+0000 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:18.659 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:18.660 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:18.661 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.657+0000 7f0ad48a9640 1 Processor -- start 2023-08-22T23:22:18.661 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.657+0000 7f0ad48a9640 1 -- start start 2023-08-22T23:22:18.661 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.657+0000 7f0ad48a9640 10 monclient: init 2023-08-22T23:22:18.661 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad00aec00 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ad00aefe0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ad00adb60 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00a7c40 con 0x7f0ad00aefe0 2023-08-22T23:22:18.662 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00a7f20 con 0x7f0ad00adb60 2023-08-22T23:22:18.663 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:18.663 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad00ae080 con 0x7f0ad00aec00 2023-08-22T23:22:18.663 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:18.663 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.658+0000 7f0ad48a9640 10 monclient(hunting): authenticate will time out at 1502.908282s 2023-08-22T23:22:18.663 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0acdd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0ad00aefe0 0x7f0ad01423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34314/0 (socket says 172.21.15.29:34314) 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0acdd9b640 1 -- 172.21.15.29:0/1033186806 learned_addr learned my addr 172.21.15.29:0/1033186806 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0ace59c640 1 --2- 172.21.15.29:0/1033186806 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0ace59c640 10 monclient(hunting): get_auth_request con 0x7f0ad00aec00 auth_method 0 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0ace59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0ace59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.659+0000 7f0ace59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:18.664 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.660+0000 7f0ace59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:18.665 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.660+0000 7f0ace59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:18.665 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.660+0000 7f0ace59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.661+0000 7f0acd59a640 1 -- 172.21.15.29:0/1033186806 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813168657 0 0) 0x7f0ad00a7c40 con 0x7f0ad00aefe0 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.661+0000 7f0acd59a640 10 monclient(hunting): _negotiate 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.661+0000 7f0acd59a640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.661+0000 7f0acd59a640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.661+0000 7f0acd59a640 10 monclient(hunting): my global_id is 4236 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.661+0000 7f0acd59a640 1 -- 172.21.15.29:0/1033186806 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0abc003520 con 0x7f0ad00aefe0 2023-08-22T23:22:18.666 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 1 -- 172.21.15.29:0/1033186806 >> v1:172.21.15.52:6789/0 conn(0x7f0ad00adb60 legacy=0x7f0ad0149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0acd59a640 1 -- 172.21.15.29:0/1033186806 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109268324 0 0) 0x7f0ad00a7f20 con 0x7f0ad00adb60 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 1 -- 172.21.15.29:0/1033186806 >> v1:172.21.15.29:6789/0 conn(0x7f0ad00aefe0 legacy=0x7f0ad01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 10 monclient: _finish_hunting 0 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 1 monclient: found mon.noname-b 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:18.667 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0ace59c640 1 -- 172.21.15.29:0/1033186806 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0065a60 con 0x7f0ad00aec00 2023-08-22T23:22:18.668 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.662+0000 7f0acd59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-08-22T23:22:18.668 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.663+0000 7f0ace59c640 1 --2- 172.21.15.29:0/1033186806 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00ac840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80097f0 tx=0x7f0ac802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c7d16a99ec3a525 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:18.669 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 1 -- 172.21.15.29:0/1033186806 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8031750 con 0x7f0ad00aec00 2023-08-22T23:22:18.669 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:18.669 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:18.669 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: dump: 2023-08-22T23:22:18.669 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:18.669 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:18.670 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: _finish_auth 0 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: _check_auth_tickets 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 1 -- 172.21.15.29:0/1033186806 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac80318b0 con 0x7f0ad00aec00 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0ad48a9640 5 monclient: authenticate success, global_id 4208 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0ad48a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:18.671 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0ad48a9640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0ad48a9640 4 set_mon_vals no callback set 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 1 -- 172.21.15.29:0/1033186806 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8031c40 con 0x7f0ad00aec00 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.667+0000 7f0acd59a640 10 monclient: dump: 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:18.672 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:18.673 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.673 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.673 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:18.673 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:18.673 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 10 monclient: shutdown 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 -- 172.21.15.29:0/1033186806 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 msgr2=0x7f0ad00ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/1033186806 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00ac840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80097f0 tx=0x7f0ac802b7f0 comp rx=0 tx=0).stop 2023-08-22T23:22:18.674 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 -- 172.21.15.29:0/1033186806 shutdown_connections 2023-08-22T23:22:18.675 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/1033186806 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00ac840 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:18.675 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 -- 172.21.15.29:0/1033186806 >> 172.21.15.29:0/1033186806 conn(0x7f0ad00b2fe0 msgr2=0x7f0ad00b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:18.675 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 -- 172.21.15.29:0/1033186806 shutdown_connections 2023-08-22T23:22:18.675 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.668+0000 7f0ad48a9640 1 -- 172.21.15.29:0/1033186806 wait complete. 2023-08-22T23:22:18.678 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.669+0000 7f0ad48a9640 10 monclient: build_initial_monmap 2023-08-22T23:22:18.678 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.669+0000 7f0ad48a9640 10 monclient: monmap: 2023-08-22T23:22:18.678 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:18.678 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:18.678 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:18.678 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 1 librados: starting msgr at 2023-08-22T23:22:18.679 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 1 librados: starting objecter 2023-08-22T23:22:18.680 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 1 Processor -- start 2023-08-22T23:22:18.680 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 1 -- start start 2023-08-22T23:22:18.680 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 1 librados: setting wanted keys 2023-08-22T23:22:18.680 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 1 librados: calling monclient init 2023-08-22T23:22:18.680 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.670+0000 7f0ad48a9640 10 monclient: init 2023-08-22T23:22:18.681 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:18.681 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:18.681 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad00adb60 0x7f0ad0136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:18.681 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad00adb60 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:18.681 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00a8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): picked mon.noname-c con 0x7f0ad00aec00 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ad00ac040 0x7f0ad01d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad00ac040 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad00aab70 con 0x7f0ad00adb60 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad01d3a20 con 0x7f0ad00ac040 2023-08-22T23:22:18.682 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:18.683 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad01d3cc0 con 0x7f0ad00aec00 2023-08-22T23:22:18.683 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:18.683 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ad48a9640 10 monclient(hunting): authenticate will time out at 1502.920683s 2023-08-22T23:22:18.683 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0ace59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad00adb60 0x7f0ad0136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:18.683 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0aced9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ad00ac040 0x7f0ad01d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.671+0000 7f0acdd9b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00a8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad00adb60 0x7f0ad0136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46528/0 (socket says 172.21.15.29:46528) 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 1 -- 172.21.15.29:0/217341363 learned_addr learned my addr 172.21.15.29:0/217341363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): get_auth_request con 0x7f0ad00adb60 auth_method 0 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:18.684 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): get_auth_request con 0x7f0ad00ac040 auth_method 0 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): get_auth_request con 0x7f0ad00aec00 auth_method 0 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:18.685 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0aced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0ace59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:18.686 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.672+0000 7f0acdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 10 monclient(hunting): handle_auth_done global_id 4240 payload 824 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 1 -- 172.21.15.29:0/217341363 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 msgr2=0x7f0ad00a8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00a8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 1 -- 172.21.15.29:0/217341363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad00adb60 msgr2=0x7f0ad0136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad00adb60 0x7f0ad0136180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:18.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 10 monclient: _finish_hunting 0 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 1 monclient: found mon.noname-b 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.675+0000 7f0aced9d640 1 -- 172.21.15.29:0/217341363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80093a0 con 0x7f0ad00ac040 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.677+0000 7f0aced9d640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ad00ac040 0x7f0ad01d41e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00adfb0 tx=0x7f0ac000cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.677+0000 7f0ac6ffd640 10 client.?.objecter ms_handle_connect 0x7f0ad00ac040 2023-08-22T23:22:18.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.677+0000 7f0ac6ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:18.689 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0003b70 con 0x7f0ad00ac040 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: dump: 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.690 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:18.691 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:18.692 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:18.692 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac0003cf0 con 0x7f0ad00ac040 2023-08-22T23:22:18.692 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:18.692 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac00052b0 con 0x7f0ad00ac040 2023-08-22T23:22:18.692 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 5 monclient: authenticate success, global_id 4240 2023-08-22T23:22:18.692 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 10 monclient: _renew_subs 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad00ab6d0 con 0x7f0ad00ac040 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ac6ffd640 10 monclient: dump: 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:18.693 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:18.694 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.694 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:18.694 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:18.694 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:18.694 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0accd99640 4 set_mon_vals no callback set 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 10 monclient: _renew_subs 2023-08-22T23:22:18.695 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:18.696 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad01d4900 con 0x7f0ad00ac040 2023-08-22T23:22:18.696 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.681+0000 7f0ad48a9640 1 librados: init done 2023-08-22T23:22:18.696 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.682+0000 7f0ad48a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:18.696 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.682+0000 7f0ad48a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:18.696 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.682+0000 7f0ad48a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:18.696 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.682+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad0074d70 con 0x7f0ad00ac040 2023-08-22T23:22:18.703 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.702+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7f0ac0005410 con 0x7f0ad00ac040 2023-08-22T23:22:18.703 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.703+0000 7f0ac6ffd640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0ab006b9a0 0x7f0ab006de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:18.704 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.703+0000 7f0ace59c640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0ab006b9a0 0x7f0ab006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:18.704 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.703+0000 7f0ace59c640 10 monclient: get_auth_request con 0x7f0ab006b9a0 auth_method 0 2023-08-22T23:22:18.704 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.703+0000 7f0ace59c640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0ab006b9a0 0x7f0ab006de60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8000c00 tx=0x7f0ac8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:18.705 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.703+0000 7f0ac6ffd640 10 client.4240.objecter ms_handle_connect 0x7f0ab006b9a0 2023-08-22T23:22:18.705 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.704+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 1925+0+0 (secure 0 0 0) 0x7f0ac005bcf0 con 0x7f0ad00ac040 2023-08-22T23:22:18.706 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.704+0000 7f0ac6ffd640 10 client.4240.objecter ms_dispatch 0x7f0ad0000b90 osd_map(12..12 src has 1..12) v4 2023-08-22T23:22:18.706 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.704+0000 7f0ac6ffd640 3 client.4240.objecter handle_osd_map got epochs [12,12] > 0 2023-08-22T23:22:18.706 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.704+0000 7f0ac6ffd640 3 client.4240.objecter handle_osd_map decoding full epoch 12 2023-08-22T23:22:18.706 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.704+0000 7f0ac6ffd640 20 client.4240.objecter dump_active .. 0 homeless 2023-08-22T23:22:18.968 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.967+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f0ac0054320 con 0x7f0ad00ac040 2023-08-22T23:22:18.968 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.967+0000 7f0ac6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:18.968 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:18.967+0000 7f0ac6ffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:19.164 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.163+0000 7f0ad48a9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:19.164 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.163+0000 7f0ad48a9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:19.164 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.163+0000 7f0ad48a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:19.165 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.163+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ad006aad0 con 0x7f0ad00ac040 2023-08-22T23:22:19.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.235+0000 7f0ac6ffd640 1 -- 172.21.15.29:0/217341363 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0ac0003e60 con 0x7f0ad00ac040 2023-08-22T23:22:19.235 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.235+0000 7f0ac6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:19.236 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.235+0000 7f0ac6ffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:19.236 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:22:19.236 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":13,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:22:18.820254+0000","last_up_change":"0.000000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:22:19.238 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.237+0000 7f0ad48a9640 10 librados: watch_flush enter 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.237+0000 7f0ad48a9640 10 librados: watch_flush exit 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.237+0000 7f0ad48a9640 20 client.4240.objecter shutdown clearing up homeless session... 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.237+0000 7f0ad48a9640 10 client.4240.objecter successfully canceled tick 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0ab006b9a0 msgr2=0x7f0ab006de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0ab006b9a0 0x7f0ab006de60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8000c00 tx=0x7f0ac8008030 comp rx=0 tx=0).stop 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 10 monclient: shutdown 2023-08-22T23:22:19.239 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ad00ac040 msgr2=0x7f0ad01d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ad00ac040 0x7f0ad01d41e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00adfb0 tx=0x7f0ac000cbe0 comp rx=0 tx=0).stop 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 shutdown_connections 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0ab006b9a0 0x7f0ab006de60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ad00ac040 0x7f0ad01d41e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0ad00aec00 0x7f0ad00a8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:19.240 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 --2- 172.21.15.29:0/217341363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad00adb60 0x7f0ad0136180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:19.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 >> 172.21.15.29:0/217341363 conn(0x7f0ad00b2fe0 msgr2=0x7f0ad00af890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:19.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 shutdown_connections 2023-08-22T23:22:19.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 -- 172.21.15.29:0/217341363 wait complete. 2023-08-22T23:22:19.241 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:19.238+0000 7f0ad48a9640 1 librados: shutdown 2023-08-22T23:22:19.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-22T23:22:19.471 INFO:tasks.ceph.osd.5.smithi052.stderr:2023-08-22T23:22:19.466+0000 ac7c640 -1 osd.5 0 waiting for initial osdmap 2023-08-22T23:22:19.544 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:22:19.539+0000 ac7c640 -1 osd.6 0 waiting for initial osdmap 2023-08-22T23:22:19.556 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:22:19.546+0000 c494640 -1 osd.4 0 waiting for initial osdmap 2023-08-22T23:22:19.871 INFO:tasks.ceph.osd.5.smithi052.stderr:2023-08-22T23:22:19.856+0000 13a60640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:19.963 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:22:19.948+0000 1325f640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:19.979 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:22:19.964+0000 13a60640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:20.764 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:22:20.759+0000 c494640 -1 osd.0 0 waiting for initial osdmap 2023-08-22T23:22:20.765 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:22:20.760+0000 c494640 -1 osd.1 0 waiting for initial osdmap 2023-08-22T23:22:20.772 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:22:20.765+0000 c494640 -1 osd.2 0 waiting for initial osdmap 2023-08-22T23:22:20.772 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:22:20.766+0000 c494640 -1 osd.3 0 waiting for initial osdmap 2023-08-22T23:22:21.200 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:22:21.183+0000 1325f640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:21.224 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:22:21.210+0000 12a5e640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:21.272 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:22:21.233+0000 1325f640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:21.273 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:22:21.251+0000 13a60640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-22T23:22:26.062 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-22T23:22:26.171 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.171+0000 7f5b81f32640 10 monclient: get_monmap_and_config 2023-08-22T23:22:26.172 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.171+0000 7f5b81f32640 10 monclient: build_initial_monmap 2023-08-22T23:22:26.172 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.171+0000 7f5b81f32640 10 monclient: monmap: 2023-08-22T23:22:26.172 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:26.172 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.172 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:22:26.171675+0000 2023-08-22T23:22:26.173 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:22:26.171675+0000 2023-08-22T23:22:26.173 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:26.173 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.173 INFO:teuthology.misc.health.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:26.173 INFO:teuthology.misc.health.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:26.173 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:26.174 INFO:teuthology.misc.health.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:26.174 INFO:teuthology.misc.health.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:26.174 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:26.175 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 1 Processor -- start 2023-08-22T23:22:26.175 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 1 -- start start 2023-08-22T23:22:26.175 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient: init 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b7c0a7bb0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b7c0a7f90 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f5b7c13eac0 0x7f5b7c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient(hunting): picked mon.noname-c con 0x7f5b7c13eac0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:26.176 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c0b0d70 con 0x7f5b7c0a7f90 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c0b0fc0 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c1420d0 con 0x7f5b7c13eac0 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.172+0000 7f5b81f32640 10 monclient(hunting): authenticate will time out at 1510.422267s 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.173+0000 7f5b7affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5b7c0a7f90 0x7f5b7c13e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34320/0 (socket says 172.21.15.29:34320) 2023-08-22T23:22:26.177 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.173+0000 7f5b7affd640 1 -- 172.21.15.29:0/3202290692 learned_addr learned my addr 172.21.15.29:0/3202290692 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:26.178 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.173+0000 7f5b7bfff640 1 --2- 172.21.15.29:0/3202290692 >> v2:172.21.15.52:3300/0 conn(0x7f5b7c13eac0 0x7f5b7c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.179 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.176+0000 7f5b7bfff640 10 monclient(hunting): get_auth_request con 0x7f5b7c13eac0 auth_method 0 2023-08-22T23:22:26.179 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.176+0000 7f5b7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.179 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.176+0000 7f5b7bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.180 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.176+0000 7f5b7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.180 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.176+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256995735 0 0) 0x7f5b7c0b0d70 con 0x7f5b7c0a7f90 2023-08-22T23:22:26.180 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): _negotiate 2023-08-22T23:22:26.180 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.180 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.180 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): my global_id is 4242 2023-08-22T23:22:26.181 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b64003520 con 0x7f5b7c0a7f90 2023-08-22T23:22:26.181 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.181 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.181 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.181 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462228525 0 0) 0x7f5b7c0b0fc0 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.181 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): _negotiate 2023-08-22T23:22:26.182 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.182 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.182 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 10 monclient(hunting): my global_id is 4243 2023-08-22T23:22:26.183 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.177+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c0b0d70 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.183 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2023-08-22T23:22:26.183 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 1 -- 172.21.15.29:0/3202290692 >> v1:172.21.15.52:6789/0 conn(0x7f5b7c0a7bb0 legacy=0x7f5b7c1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.184 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 1 -- 172.21.15.29:0/3202290692 >> v1:172.21.15.29:6789/0 conn(0x7f5b7c0a7f90 legacy=0x7f5b7c13e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.184 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:26.184 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 1 monclient: found mon.noname-c 2023-08-22T23:22:26.184 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:26.185 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.185 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.179+0000 7f5b7bfff640 1 -- 172.21.15.29:0/3202290692 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0a9b20 con 0x7f5b7c13eac0 2023-08-22T23:22:26.185 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.182+0000 7f5b7bfff640 1 --2- 172.21.15.29:0/3202290692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c13eac0 0x7f5b7c0b12a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009dc0 tx=0x7f5b7402bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bbb42ac17404bef server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:26.188 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7402ccd0 con 0x7f5b7c13eac0 2023-08-22T23:22:26.188 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.188 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: dump: 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.189 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:26.190 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7402ce30 con 0x7f5b7c13eac0 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b81f32640 5 monclient: authenticate success, global_id 4246 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b81f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 1 -- 172.21.15.29:0/3202290692 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74034660 con 0x7f5b7c13eac0 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b81f32640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b81f32640 4 set_mon_vals no callback set 2023-08-22T23:22:26.191 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.186+0000 7f5b7a7fc640 10 monclient: dump: 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.192 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 10 monclient: shutdown 2023-08-22T23:22:26.193 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 -- 172.21.15.29:0/3202290692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c13eac0 msgr2=0x7f5b7c0b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3202290692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c13eac0 0x7f5b7c0b12a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009dc0 tx=0x7f5b7402bf20 comp rx=0 tx=0).stop 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 -- 172.21.15.29:0/3202290692 shutdown_connections 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3202290692 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c13eac0 0x7f5b7c0b12a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 -- 172.21.15.29:0/3202290692 >> 172.21.15.29:0/3202290692 conn(0x7f5b7c0b6920 msgr2=0x7f5b7c0b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 -- 172.21.15.29:0/3202290692 shutdown_connections 2023-08-22T23:22:26.194 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.187+0000 7f5b81f32640 1 -- 172.21.15.29:0/3202290692 wait complete. 2023-08-22T23:22:26.197 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 10 monclient: build_initial_monmap 2023-08-22T23:22:26.197 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 10 monclient: monmap: 2023-08-22T23:22:26.197 INFO:teuthology.misc.health.smithi029.stderr:epoch 0 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:created 0.000000 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:26.198 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:26.199 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:26.199 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 1 librados: starting msgr at 2023-08-22T23:22:26.199 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 1 librados: starting objecter 2023-08-22T23:22:26.199 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 1 Processor -- start 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 1 -- start start 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 1 librados: setting wanted keys 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 1 librados: calling monclient init 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.188+0000 7f5b81f32640 10 monclient: init 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 0x7f5b7c13d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.200 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7c0a7bb0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7c0a7f90 0x7f5b7c1424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): picked mon.noname-a con 0x7f5b7c0a7f90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b7c13eac0 0x7f5b7c142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): picked mon.noname-c con 0x7f5b7c13eac0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c143500 con 0x7f5b7c0a7f90 2023-08-22T23:22:26.201 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0b61c0 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c1448f0 con 0x7f5b7c13eac0 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.189+0000 7f5b81f32640 10 monclient(hunting): authenticate will time out at 1510.438993s 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 0x7f5b7c13d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7bfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b7c13eac0 0x7f5b7c142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.202 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7c0a7f90 0x7f5b7c1424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 0x7f5b7c13d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:41822/0 (socket says 172.21.15.29:41822) 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 1 -- 172.21.15.29:0/3247728442 learned_addr learned my addr 172.21.15.29:0/3247728442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): get_auth_request con 0x7f5b7c0a7bb0 auth_method 0 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7affd640 10 monclient(hunting): get_auth_request con 0x7f5b7c0a7f90 auth_method 0 2023-08-22T23:22:26.203 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7affd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7affd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7bfff640 10 monclient(hunting): get_auth_request con 0x7f5b7c13eac0 auth_method 0 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.204 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.190+0000 7f5b7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.191+0000 7f5b7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.191+0000 7f5b7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.191+0000 7f5b7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.191+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.191+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.205 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.191+0000 7f5b7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 10 monclient(hunting): handle_auth_done global_id 4249 payload 824 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 1 -- 172.21.15.29:0/3247728442 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b7c13eac0 msgr2=0x7f5b7c142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b7c13eac0 0x7f5b7c142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 1 -- 172.21.15.29:0/3247728442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7c0a7f90 msgr2=0x7f5b7c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7c0a7f90 0x7f5b7c1424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:26.206 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 1 monclient: found mon.noname-b 2023-08-22T23:22:26.207 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:26.207 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.207 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.193+0000 7f5b7b7fe640 1 -- 172.21.15.29:0/3247728442 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74009a70 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.207 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.196+0000 7f5b7b7fe640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 0x7f5b7c13d460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c500 tx=0x7f5b6c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:26.207 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.196+0000 7f5b63fff640 10 client.?.objecter ms_handle_connect 0x7f5b7c0a7bb0 2023-08-22T23:22:26.207 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.196+0000 7f5b63fff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:26.208 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c0052d0 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.208 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.209 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:26.209 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: dump: 2023-08-22T23:22:26.209 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:26.209 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.210 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.210 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.210 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.210 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.211 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.211 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.211 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.212 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:26.212 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: _finish_auth 0 2023-08-22T23:22:26.212 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: _check_auth_tickets 2023-08-22T23:22:26.212 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:26.213 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c005450 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.213 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:26.213 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c012870 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.214 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 5 monclient: authenticate success, global_id 4249 2023-08-22T23:22:26.214 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 10 monclient: _renew_subs 2023-08-22T23:22:26.214 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.214 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c143f70 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.215 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.215 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:26.215 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b63fff640 10 monclient: dump: 2023-08-22T23:22:26.216 INFO:teuthology.misc.health.smithi029.stderr:epoch 1 2023-08-22T23:22:26.216 INFO:teuthology.misc.health.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.216 INFO:teuthology.misc.health.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.216 INFO:teuthology.misc.health.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.217 INFO:teuthology.misc.health.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.217 INFO:teuthology.misc.health.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.217 INFO:teuthology.misc.health.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.218 INFO:teuthology.misc.health.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.218 INFO:teuthology.misc.health.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.218 INFO:teuthology.misc.health.smithi029.stderr: 2023-08-22T23:22:26.218 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b79ffb640 4 set_mon_vals no callback set 2023-08-22T23:22:26.219 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:26.219 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 10 monclient: _renew_subs 2023-08-22T23:22:26.219 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.220 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c0b6610 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.220 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.200+0000 7f5b81f32640 1 librados: init done 2023-08-22T23:22:26.220 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.201+0000 7f5b81f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:26.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.201+0000 7f5b81f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:26.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.201+0000 7f5b81f32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.221 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.201+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c0b6330 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.221+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7f5b6c0129d0 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.222+0000 7f5b63fff640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f5b6806b9a0 0x7f5b6806de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.223 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.222+0000 7f5b7affd640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f5b6806b9a0 0x7f5b6806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.222+0000 7f5b7affd640 10 monclient: get_auth_request con 0x7f5b6806b9a0 auth_method 0 2023-08-22T23:22:26.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.222+0000 7f5b63fff640 10 client.4249.objecter ms_handle_connect 0x7f5b6806b9a0 2023-08-22T23:22:26.224 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.222+0000 7f5b7affd640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f5b6806b9a0 0x7f5b6806de60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0b2830 tx=0x7f5b70007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:26.471 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.471+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b6c05bad0 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.472 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.471+0000 7f5b63fff640 10 client.4249.objecter ms_dispatch 0x7f5b7c000b90 osd_map(18..18 src has 1..18) v4 2023-08-22T23:22:26.472 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.471+0000 7f5b63fff640 3 client.4249.objecter handle_osd_map got epochs [18,18] > 0 2023-08-22T23:22:26.472 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.471+0000 7f5b63fff640 3 client.4249.objecter handle_osd_map decoding full epoch 18 2023-08-22T23:22:26.472 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.471+0000 7f5b63fff640 20 client.4249.objecter dump_active .. 0 homeless 2023-08-22T23:22:26.484 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.483+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f5b6c054100 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.484 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.483+0000 7f5b63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:26.484 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.483+0000 7f5b63fff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:26.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.686+0000 7f5b81f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:26.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.686+0000 7f5b81f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:26.687 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.686+0000 7f5b81f32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.688 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.686+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b7c074d70 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.782+0000 7f5b63fff640 1 -- 172.21.15.29:0/3247728442 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12815 (secure 0 0 0) 0x7f5b6c057720 con 0x7f5b7c0a7bb0 2023-08-22T23:22:26.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.782+0000 7f5b63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:26.783 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.782+0000 7f5b63fff640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:26.784 INFO:teuthology.misc.health.smithi029.stdout: 2023-08-22T23:22:26.784 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":18,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:22:24.829126+0000","last_up_change":"2023-08-22T23:22:22.559235+0000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-22T23:22:20.507839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6809","nonce":1473840652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6811","nonce":1473840652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6816","nonce":1473840652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6813","nonce":1473840652}]},"public_addr":"172.21.15.29:6809/1473840652","cluster_addr":"172.21.15.29:6811/1473840652","heartbeat_back_addr":"172.21.15.29:6816/1473840652","heartbeat_front_addr":"172.21.15.29:6813/1473840652","state":["exists","up"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6825","nonce":2619884743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6827","nonce":2619884743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6831","nonce":2619884743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6829","nonce":2619884743}]},"public_addr":"172.21.15.29:6825/2619884743","cluster_addr":"172.21.15.29:6827/2619884743","heartbeat_back_addr":"172.21.15.29:6831/2619884743","heartbeat_front_addr":"172.21.15.29:6829/2619884743","state":["exists","up"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6801","nonce":1029571786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6803","nonce":1029571786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6807","nonce":1029571786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6805","nonce":1029571786}]},"public_addr":"172.21.15.29:6801/1029571786","cluster_addr":"172.21.15.29:6803/1029571786","heartbeat_back_addr":"172.21.15.29:6807/1029571786","heartbeat_front_addr":"172.21.15.29:6805/1029571786","state":["exists","up"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6817","nonce":4227673673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6819","nonce":4227673673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6823","nonce":4227673673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6821","nonce":4227673673}]},"public_addr":"172.21.15.29:6817/4227673673","cluster_addr":"172.21.15.29:6819/4227673673","heartbeat_back_addr":"172.21.15.29:6823/4227673673","heartbeat_front_addr":"172.21.15.29:6821/4227673673","state":["exists","up"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6803","nonce":2157117708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6805","nonce":2157117708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6809","nonce":2157117708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6807","nonce":2157117708}]},"public_addr":"172.21.15.52:6803/2157117708","cluster_addr":"172.21.15.52:6805/2157117708","heartbeat_back_addr":"172.21.15.52:6809/2157117708","heartbeat_front_addr":"172.21.15.52:6807/2157117708","state":["exists","up"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6827","nonce":4164832567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6829","nonce":4164832567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6833","nonce":4164832567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6831","nonce":4164832567}]},"public_addr":"172.21.15.52:6827/4164832567","cluster_addr":"172.21.15.52:6829/4164832567","heartbeat_back_addr":"172.21.15.52:6833/4164832567","heartbeat_front_addr":"172.21.15.52:6831/4164832567","state":["exists","up"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6819","nonce":2556931765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6821","nonce":2556931765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6825","nonce":2556931765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6823","nonce":2556931765}]},"public_addr":"172.21.15.52:6819/2556931765","cluster_addr":"172.21.15.52:6821/2556931765","heartbeat_back_addr":"172.21.15.52:6825/2556931765","heartbeat_front_addr":"172.21.15.52:6823/2556931765","state":["exists","up"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6811","nonce":3454580439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6813","nonce":3454580439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6817","nonce":3454580439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6815","nonce":3454580439}]},"public_addr":"172.21.15.52:6811/3454580439","cluster_addr":"172.21.15.52:6813/3454580439","heartbeat_back_addr":"172.21.15.52:6817/3454580439","heartbeat_front_addr":"172.21.15.52:6815/3454580439","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:19.045553+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.775383+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.732376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.515362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.723819+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.012554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.629140+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.069717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:22:26.785 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 10 librados: watch_flush enter 2023-08-22T23:22:26.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 10 librados: watch_flush exit 2023-08-22T23:22:26.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 20 client.4249.objecter shutdown clearing up homeless session... 2023-08-22T23:22:26.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 10 client.4249.objecter successfully canceled tick 2023-08-22T23:22:26.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f5b6806b9a0 msgr2=0x7f5b6806de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f5b6806b9a0 0x7f5b6806de60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0b2830 tx=0x7f5b70007d30 comp rx=0 tx=0).stop 2023-08-22T23:22:26.786 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 10 monclient: shutdown 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 msgr2=0x7f5b7c13d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 0x7f5b7c13d460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c500 tx=0x7f5b6c00c9c0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 shutdown_connections 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f5b6806b9a0 0x7f5b6806de60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f5b7c13eac0 0x7f5b7c142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.787 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7c0a7f90 0x7f5b7c1424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.788 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 --2- 172.21.15.29:0/3247728442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b7c0a7bb0 0x7f5b7c13d460 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.788 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 >> 172.21.15.29:0/3247728442 conn(0x7f5b7c0b6920 msgr2=0x7f5b7c0b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:26.788 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.784+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 shutdown_connections 2023-08-22T23:22:26.788 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.785+0000 7f5b81f32640 1 -- 172.21.15.29:0/3247728442 wait complete. 2023-08-22T23:22:26.788 INFO:teuthology.misc.health.smithi029.stderr:2023-08-22T23:22:26.785+0000 7f5b81f32640 1 librados: shutdown 2023-08-22T23:22:26.795 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-08-22T23:22:26.795 INFO:tasks.ceph:Creating RBD pool 2023-08-22T23:22:26.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool create rbd 8 2023-08-22T23:22:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.906+0000 7f6256398640 10 monclient: get_monmap_and_config 2023-08-22T23:22:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.906+0000 7f6256398640 10 monclient: build_initial_monmap 2023-08-22T23:22:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.906+0000 7f6256398640 10 monclient: monmap: 2023-08-22T23:22:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:26.906860+0000 2023-08-22T23:22:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:26.906860+0000 2023-08-22T23:22:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:26.909 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 1 Processor -- start 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 1 -- start start 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 10 monclient: init 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f6250144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 10 monclient(hunting): picked mon.noname-b con 0x7f62501448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.907+0000 7f6256398640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62500add90 0x7f6250145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): picked mon.noname-a con 0x7f62500add90 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f625013e190 0x7f625013e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): picked mon.noname-c con 0x7f625013e190 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625013eb50 con 0x7f62500add90 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500b12d0 con 0x7f625013e190 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62500afae0 con 0x7f62501448e0 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6256398640 10 monclient(hunting): authenticate will time out at 1511.157504s 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62500add90 0x7f6250145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624ffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f6250144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f6254935640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f625013e190 0x7f625013e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62500add90 0x7f6250145200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46550/0 (socket says 172.21.15.29:46550) 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 1 -- 172.21.15.29:0/3829758811 learned_addr learned my addr 172.21.15.29:0/3829758811 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 10 monclient(hunting): get_auth_request con 0x7f62500add90 auth_method 0 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.908+0000 7f624f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): get_auth_request con 0x7f62501448e0 auth_method 0 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): get_auth_request con 0x7f625013e190 auth_method 0 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f6254935640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.909+0000 7f624ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 10 monclient(hunting): handle_auth_done global_id 4252 payload 274 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 1 -- 172.21.15.29:0/3829758811 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 msgr2=0x7f6250144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 1 --2- 172.21.15.29:0/3829758811 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f6250144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 1 -- 172.21.15.29:0/3829758811 >> v2:172.21.15.29:3300/0 conn(0x7f62500add90 msgr2=0x7f6250145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 1 --2- 172.21.15.29:0/3829758811 >> v2:172.21.15.29:3300/0 conn(0x7f62500add90 0x7f6250145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 10 monclient: _finish_hunting 0 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 1 monclient: found mon.noname-c 2023-08-22T23:22:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.911+0000 7f6254935640 1 -- 172.21.15.29:0/3829758811 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500afdc0 con 0x7f625013e190 2023-08-22T23:22:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.913+0000 7f6254935640 1 --2- 172.21.15.29:0/3829758811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f625013e610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62500ad970 tx=0x7f624802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bebcf3755336514e server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 1 -- 172.21.15.29:0/3829758811 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624802cbd0 con 0x7f625013e190 2023-08-22T23:22:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: dump: 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: _finish_auth 0 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 1 -- 172.21.15.29:0/3829758811 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624802cd30 con 0x7f625013e190 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 5 monclient: authenticate success, global_id 4252 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 4 set_mon_vals no callback set 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 1 -- 172.21.15.29:0/3829758811 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624802cea0 con 0x7f625013e190 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f624effd640 10 monclient: dump: 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.924 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 10 monclient: shutdown 2023-08-22T23:22:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 -- 172.21.15.29:0/3829758811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 msgr2=0x7f625013e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 --2- 172.21.15.29:0/3829758811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f625013e610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62500ad970 tx=0x7f624802be20 comp rx=0 tx=0).stop 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 -- 172.21.15.29:0/3829758811 shutdown_connections 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 --2- 172.21.15.29:0/3829758811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f625013e610 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 --2- 172.21.15.29:0/3829758811 >> v2:172.21.15.29:3300/0 conn(0x7f62500add90 0x7f6250145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 --2- 172.21.15.29:0/3829758811 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f6250144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 -- 172.21.15.29:0/3829758811 >> 172.21.15.29:0/3829758811 conn(0x7f6250065ed0 msgr2=0x7f62500b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.917+0000 7f6256398640 1 -- 172.21.15.29:0/3829758811 shutdown_connections 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.918+0000 7f6256398640 1 -- 172.21.15.29:0/3829758811 wait complete. 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.918+0000 7f6256398640 10 monclient: build_initial_monmap 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.918+0000 7f6256398640 10 monclient: monmap: 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.918+0000 7f6256398640 1 librados: starting msgr at 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.918+0000 7f6256398640 1 librados: starting objecter 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 Processor -- start 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 -- start start 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 librados: setting wanted keys 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 librados: calling monclient init 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient: init 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62500add90 0x7f625013abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): picked mon.noname-a con 0x7f62500add90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f62501411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): picked mon.noname-b con 0x7f625013e190 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f62501d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): picked mon.noname-c con 0x7f62501448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250141870 con 0x7f62500add90 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250141d00 con 0x7f625013e190 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625013b6c0 con 0x7f62501448e0 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.919+0000 7f6256398640 10 monclient(hunting): authenticate will time out at 1511.169290s 2023-08-22T23:22:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f62501d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f624f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f62501411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f624ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62500add90 0x7f625013abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f62501d60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:56208/0 (socket says 172.21.15.29:56208) 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 1 -- 172.21.15.29:0/2775624922 learned_addr learned my addr 172.21.15.29:0/2775624922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 10 monclient(hunting): get_auth_request con 0x7f62501448e0 auth_method 0 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f6254935640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f624f7fe640 10 monclient(hunting): get_auth_request con 0x7f625013e190 auth_method 0 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f624f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f624f7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.920+0000 7f624f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): get_auth_request con 0x7f62500add90 auth_method 0 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f6254935640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f6254935640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f6254935640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.921+0000 7f624ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2023-08-22T23:22:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 1 -- 172.21.15.29:0/2775624922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 msgr2=0x7f62501d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f62501d60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 1 -- 172.21.15.29:0/2775624922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62500add90 msgr2=0x7f625013abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62500add90 0x7f625013abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 1 monclient: found mon.noname-b 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.924+0000 7f624f7fe640 1 -- 172.21.15.29:0/2775624922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6248009a70 con 0x7f625013e190 2023-08-22T23:22:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.926+0000 7f624f7fe640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f62501411c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f624400e870 tx=0x7f624400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.926+0000 7f624cff9640 10 client.?.objecter ms_handle_connect 0x7f625013e190 2023-08-22T23:22:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.926+0000 7f624cff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624400ee60 con 0x7f625013e190 2023-08-22T23:22:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: dump: 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.940 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624400c940 con 0x7f625013e190 2023-08-22T23:22:26.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f6256398640 5 monclient: authenticate success, global_id 4255 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f6256398640 10 monclient: _renew_subs 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f6256398640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6250140d80 con 0x7f625013e190 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624400ccd0 con 0x7f625013e190 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f624cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:26.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f624cff9640 10 monclient: dump: 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:26.943 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.929+0000 7f624e7fc640 4 set_mon_vals no callback set 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f6256398640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f6256398640 10 monclient: _renew_subs 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f6256398640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62501d67d0 con 0x7f625013e190 2023-08-22T23:22:26.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.930+0000 7f6256398640 1 librados: init done 2023-08-22T23:22:26.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.931+0000 7f6256398640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:26.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.931+0000 7f6256398640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:26.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.931+0000 7f6256398640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:26.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.931+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6250066cb0 con 0x7f625013e190 2023-08-22T23:22:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.953+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7f6244013470 con 0x7f625013e190 2023-08-22T23:22:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.953+0000 7f624cff9640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f623c06b950 0x7f623c06de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.953+0000 7f624ffff640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f623c06b950 0x7f623c06de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.953+0000 7f624ffff640 10 monclient: get_auth_request con 0x7f623c06b950 auth_method 0 2023-08-22T23:22:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.954+0000 7f624ffff640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f623c06b950 0x7f623c06de10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6240009ac0 tx=0x7f6240009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.954+0000 7f624cff9640 10 client.4255.objecter ms_handle_connect 0x7f623c06b950 2023-08-22T23:22:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.957+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f624400ce30 con 0x7f625013e190 2023-08-22T23:22:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.957+0000 7f624cff9640 10 client.4255.objecter ms_dispatch 0x7f6250000b90 osd_map(18..18 src has 1..18) v4 2023-08-22T23:22:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.957+0000 7f624cff9640 3 client.4255.objecter handle_osd_map got epochs [18,18] > 0 2023-08-22T23:22:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.957+0000 7f624cff9640 3 client.4255.objecter handle_osd_map decoding full epoch 18 2023-08-22T23:22:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:26.957+0000 7f624cff9640 20 client.4255.objecter dump_active .. 0 homeless 2023-08-22T23:22:27.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.215+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f62440544d0 con 0x7f625013e190 2023-08-22T23:22:27.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.215+0000 7f624cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.215+0000 7f624cff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.456+0000 7f6256398640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-08-22T23:22:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.456+0000 7f6256398640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-08-22T23:22:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.456+0000 7f6256398640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:27.456+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f62500b6b30 con 0x7f625013e190 2023-08-22T23:22:28.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.637+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f62440548a0 con 0x7f625013e190 2023-08-22T23:22:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.710+0000 7f624cff9640 1 -- 172.21.15.29:0/2775624922 <== mon.1 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f6244057f60 con 0x7f625013e190 2023-08-22T23:22:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.710+0000 7f624cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-08-22T23:22:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.710+0000 7f624cff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-08-22T23:22:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2023-08-22T23:22:28.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.712+0000 7f6256398640 10 librados: watch_flush enter 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.712+0000 7f6256398640 10 librados: watch_flush exit 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.712+0000 7f6256398640 20 client.4255.objecter shutdown clearing up homeless session... 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 10 client.4255.objecter successfully canceled tick 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f623c06b950 msgr2=0x7f623c06de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f623c06b950 0x7f623c06de10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6240009ac0 tx=0x7f6240009190 comp rx=0 tx=0).stop 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 10 monclient: shutdown 2023-08-22T23:22:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 msgr2=0x7f62501411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f62501411c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f624400e870 tx=0x7f624400ed30 comp rx=0 tx=0).stop 2023-08-22T23:22:28.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 shutdown_connections 2023-08-22T23:22:28.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f623c06b950 0x7f623c06de10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f62501448e0 0x7f62501d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f625013e190 0x7f62501411c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.713+0000 7f6256398640 1 --2- 172.21.15.29:0/2775624922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62500add90 0x7f625013abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.714+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 >> 172.21.15.29:0/2775624922 conn(0x7f6250065ed0 msgr2=0x7f6250144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:28.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.714+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 shutdown_connections 2023-08-22T23:22:28.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.714+0000 7f6256398640 1 -- 172.21.15.29:0/2775624922 wait complete. 2023-08-22T23:22:28.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.714+0000 7f6256398640 1 librados: shutdown 2023-08-22T23:22:28.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-08-22T23:22:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.847+0000 7fcf0312b640 10 monclient: get_monmap_and_config 2023-08-22T23:22:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.847+0000 7fcf0312b640 10 monclient: build_initial_monmap 2023-08-22T23:22:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.847+0000 7fcf0312b640 10 monclient: monmap: 2023-08-22T23:22:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:28.847594+0000 2023-08-22T23:22:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:28.847594+0000 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:28.850 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 Processor -- start 2023-08-22T23:22:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 -- start start 2023-08-22T23:22:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient: init 2023-08-22T23:22:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a7c10 0x7fcefc13e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): picked mon.noname-b con 0x7fcefc0a7c10 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcefc13eb80 0x7fcefc144a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): picked mon.noname-a con 0x7fcefc13eb80 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcefc0a8e90 0x7fcefc144f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): picked mon.noname-c con 0x7fcefc0a8e90 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:28.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0a9c80 con 0x7fcefc13eb80 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0b0dd0 con 0x7fcefc0a8e90 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b1020 con 0x7fcefc0a7c10 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.848+0000 7fcf0312b640 10 monclient(hunting): authenticate will time out at 1513.098230s 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcefc13eb80 0x7fcefc144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcf016c8640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcefc0a8e90 0x7fcefc144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcf00ec7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a7c10 0x7fcefc13e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcefc13eb80 0x7fcefc144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33248/0 (socket says 172.21.15.29:33248) 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 1 -- 172.21.15.29:0/3341385882 learned_addr learned my addr 172.21.15.29:0/3341385882 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 10 monclient(hunting): get_auth_request con 0x7fcefc13eb80 auth_method 0 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:28.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcefbfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcf016c8640 10 monclient(hunting): get_auth_request con 0x7fcefc0a8e90 auth_method 0 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcf016c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcf016c8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.849+0000 7fcf016c8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): get_auth_request con 0x7fcefc0a7c10 auth_method 0 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcefbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcefbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcefbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf016c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf016c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:28.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf016c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.850+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.851+0000 7fcefbfff640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 1 -- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a7c10 msgr2=0x7fcefc13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 1 --2- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a7c10 0x7fcefc13e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 1 -- 172.21.15.29:0/3341385882 >> v2:172.21.15.52:3300/0 conn(0x7fcefc0a8e90 msgr2=0x7fcefc144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 1 --2- 172.21.15.29:0/3341385882 >> v2:172.21.15.52:3300/0 conn(0x7fcefc0a8e90 0x7fcefc144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 1 monclient: found mon.noname-a 2023-08-22T23:22:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.852+0000 7fcefbfff640 1 -- 172.21.15.29:0/3341385882 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0a9e60 con 0x7fcefc13eb80 2023-08-22T23:22:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.854+0000 7fcefbfff640 1 --2- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc13eb80 0x7fcefc144a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0a9ab0 tx=0x7fcee802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edd6e715b87bb46 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 1 -- 172.21.15.29:0/3341385882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee80317e0 con 0x7fcefc13eb80 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: dump: 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 1 -- 172.21.15.29:0/3341385882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee8031940 con 0x7fcefc13eb80 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 1 -- 172.21.15.29:0/3341385882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8031cd0 con 0x7fcefc13eb80 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcefb7fe640 10 monclient: dump: 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcf0312b640 5 monclient: authenticate success, global_id 4260 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcf0312b640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.858+0000 7fcf0312b640 4 set_mon_vals no callback set 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 10 monclient: shutdown 2023-08-22T23:22:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 -- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc13eb80 msgr2=0x7fcefc144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 --2- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc13eb80 0x7fcefc144a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0a9ab0 tx=0x7fcee802b880 comp rx=0 tx=0).stop 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 -- 172.21.15.29:0/3341385882 shutdown_connections 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 --2- 172.21.15.29:0/3341385882 >> v2:172.21.15.52:3300/0 conn(0x7fcefc0a8e90 0x7fcefc144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 --2- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc13eb80 0x7fcefc144a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 --2- 172.21.15.29:0/3341385882 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a7c10 0x7fcefc13e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 -- 172.21.15.29:0/3341385882 >> 172.21.15.29:0/3341385882 conn(0x7fcefc0b69b0 msgr2=0x7fcefc0b6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 -- 172.21.15.29:0/3341385882 shutdown_connections 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.859+0000 7fcf0312b640 1 -- 172.21.15.29:0/3341385882 wait complete. 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 10 monclient: build_initial_monmap 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 10 monclient: monmap: 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 1 librados: starting msgr at 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 1 librados: starting objecter 2023-08-22T23:22:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 1 Processor -- start 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 1 -- start start 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 1 librados: setting wanted keys 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 1 librados: calling monclient init 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.860+0000 7fcf0312b640 10 monclient: init 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 0x7fcefc139d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): picked mon.noname-a con 0x7fcefc0a7c10 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a8e90 0x7fcefc13a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): picked mon.noname-c con 0x7fcefc0a8e90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcefc13eb80 0x7fcefc134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): picked mon.noname-b con 0x7fcefc13eb80 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc139000 con 0x7fcefc0a7c10 2023-08-22T23:22:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc1391e0 con 0x7fcefc13eb80 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc1419b0 con 0x7fcefc0a8e90 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.861+0000 7fcf0312b640 10 monclient(hunting): authenticate will time out at 1513.111121s 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 0x7fcefc139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf016c8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcefc13eb80 0x7fcefc134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcefbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a8e90 0x7fcefc13a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 0x7fcefc139d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33250/0 (socket says 172.21.15.29:33250) 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 1 -- 172.21.15.29:0/790425954 learned_addr learned my addr 172.21.15.29:0/790425954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 10 monclient(hunting): get_auth_request con 0x7fcefc0a7c10 auth_method 0 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf00ec7640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf016c8640 10 monclient(hunting): get_auth_request con 0x7fcefc13eb80 auth_method 0 2023-08-22T23:22:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf016c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf016c8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcf016c8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcefbfff640 10 monclient(hunting): get_auth_request con 0x7fcefc0a8e90 auth_method 0 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcefbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcefbfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.862+0000 7fcefbfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcf016c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcf016c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcf016c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcefbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcefbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.863+0000 7fcefbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.865+0000 7fcf00ec7640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2023-08-22T23:22:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 1 -- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a8e90 msgr2=0x7fcefc13a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a8e90 0x7fcefc13a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 1 -- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcefc13eb80 msgr2=0x7fcefc134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcefc13eb80 0x7fcefc134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 10 monclient: _finish_hunting 0 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 1 monclient: found mon.noname-a 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.866+0000 7fcf00ec7640 1 -- 172.21.15.29:0/790425954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee80094d0 con 0x7fcefc0a7c10 2023-08-22T23:22:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.868+0000 7fcf00ec7640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 0x7fcefc139d80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e890 tx=0x7fcef000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.868+0000 7fcef97fa640 10 client.?.objecter ms_handle_connect 0x7fcefc0a7c10 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.868+0000 7fcef97fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef000c940 con 0x7fcefc0a7c10 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: dump: 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:28.882 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: _finish_auth 0 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: _check_auth_tickets 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef000caa0 con 0x7fcefc0a7c10 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef000ce30 con 0x7fcefc0a7c10 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:28.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcef97fa640 10 monclient: dump: 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:28.884 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 5 monclient: authenticate success, global_id 4263 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcefaffd640 4 set_mon_vals no callback set 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 10 monclient: _renew_subs 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc145620 con 0x7fcefc0a7c10 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 10 monclient: _renew_subs 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc13bcd0 con 0x7fcefc0a7c10 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.873+0000 7fcf0312b640 1 librados: init done 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.874+0000 7fcf0312b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.874+0000 7fcf0312b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:28.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.874+0000 7fcf0312b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:28.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:28.874+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc0a7760 con 0x7fcefc0a7c10 2023-08-22T23:22:29.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.174+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcef0013470 con 0x7fcefc0a7c10 2023-08-22T23:22:29.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.174+0000 7fcef97fa640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcee006ba30 0x7fcee006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:29.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.174+0000 7fcefbfff640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcee006ba30 0x7fcee006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:29.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.174+0000 7fcefbfff640 10 monclient: get_auth_request con 0x7fcee006ba30 auth_method 0 2023-08-22T23:22:29.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.175+0000 7fcefbfff640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcee006ba30 0x7fcee006def0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcee8036000 tx=0x7fcee803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:29.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.175+0000 7fcef97fa640 10 client.4263.objecter ms_handle_connect 0x7fcee006ba30 2023-08-22T23:22:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7fcef005bfc0 con 0x7fcefc0a7c10 2023-08-22T23:22:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 10 client.4263.objecter ms_dispatch 0x7fcefc000b90 osd_map(19..19 src has 1..19) v4 2023-08-22T23:22:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 3 client.4263.objecter handle_osd_map got epochs [19,19] > 0 2023-08-22T23:22:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 3 client.4263.objecter handle_osd_map decoding full epoch 19 2023-08-22T23:22:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 20 client.4263.objecter dump_active .. 0 homeless 2023-08-22T23:22:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fcef00545f0 con 0x7fcefc0a7c10 2023-08-22T23:22:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:29.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.192+0000 7fcef97fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.422+0000 7fcf0312b640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-08-22T23:22:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.422+0000 7fcf0312b640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-08-22T23:22:29.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.422+0000 7fcf0312b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:29.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.422+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcefc066c30 con 0x7fcefc0a7c10 2023-08-22T23:22:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.627+0000 7fcef97fa640 1 -- 172.21.15.29:0/790425954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7fcef00210c0 con 0x7fcefc0a7c10 2023-08-22T23:22:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.627+0000 7fcef97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-08-22T23:22:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.627+0000 7fcef97fa640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-08-22T23:22:29.628 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rbd' on pool 'rbd' 2023-08-22T23:22:29.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.629+0000 7fcf0312b640 10 librados: watch_flush enter 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.629+0000 7fcf0312b640 10 librados: watch_flush exit 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.629+0000 7fcf0312b640 20 client.4263.objecter shutdown clearing up homeless session... 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.629+0000 7fcf0312b640 10 client.4263.objecter successfully canceled tick 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcee006ba30 msgr2=0x7fcee006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcee006ba30 0x7fcee006def0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcee8036000 tx=0x7fcee803d040 comp rx=0 tx=0).stop 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 10 monclient: shutdown 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:29.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 msgr2=0x7fcefc139d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 0x7fcefc139d80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e890 tx=0x7fcef000ed50 comp rx=0 tx=0).stop 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 shutdown_connections 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fcee006ba30 0x7fcee006def0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcefc13eb80 0x7fcefc134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fcefc0a8e90 0x7fcefc13a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 --2- 172.21.15.29:0/790425954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc0a7c10 0x7fcefc139d80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 >> 172.21.15.29:0/790425954 conn(0x7fcefc0b69b0 msgr2=0x7fcefc0b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 shutdown_connections 2023-08-22T23:22:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 -- 172.21.15.29:0/790425954 wait complete. 2023-08-22T23:22:29.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.630+0000 7fcf0312b640 1 librados: shutdown 2023-08-22T23:22:29.647 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-08-22T23:22:29.648 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-08-22T23:22:29.648 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-08-22T23:22:29.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-08-22T23:22:29.649 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-08-22T23:22:29.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.759+0000 7f16f2806640 10 monclient: get_monmap_and_config 2023-08-22T23:22:29.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.759+0000 7f16f2806640 10 monclient: build_initial_monmap 2023-08-22T23:22:29.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.759+0000 7f16f2806640 10 monclient: monmap: 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:29.760198+0000 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:29.760198+0000 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:29.762 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:29.762 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:29.762 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:29.762 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:29.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.760+0000 7f16f2806640 1 Processor -- start 2023-08-22T23:22:29.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.760+0000 7f16f2806640 1 -- start start 2023-08-22T23:22:29.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.760+0000 7f16f2806640 10 monclient: init 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16ec0a7b90 0x7f16ec13e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): picked mon.noname-a con 0x7f16ec0a7b90 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16ec13ead0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0b1380 0x7f16ec14ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): picked mon.noname-b con 0x7f16ec0b1380 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec0a9b00 con 0x7f16ec13ead0 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec0a9de0 con 0x7f16ec0a7b90 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0aa0c0 con 0x7f16ec0b1380 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:29.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16f2806640 10 monclient(hunting): authenticate will time out at 1514.010773s 2023-08-22T23:22:29.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.761+0000 7f16eb7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f16ec13ead0 0x7f16ec1417e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44528/0 (socket says 172.21.15.29:44528) 2023-08-22T23:22:29.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16eb7fe640 1 -- 172.21.15.29:0/2821207078 learned_addr learned my addr 172.21.15.29:0/2821207078 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:29.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16f0da3640 1 --2- 172.21.15.29:0/2821207078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0b1380 0x7f16ec14ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:29.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16ebfff640 1 --2- 172.21.15.29:0/2821207078 >> v2:172.21.15.29:3300/0 conn(0x7f16ec0a7b90 0x7f16ec13e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:29.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16ebfff640 10 monclient(hunting): get_auth_request con 0x7f16ec0a7b90 auth_method 0 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16ebfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16f0da3640 10 monclient(hunting): get_auth_request con 0x7f16ec0b1380 auth_method 0 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16f0da3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16f0da3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:29.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.762+0000 7f16f0da3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:29.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.763+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:29.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.763+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:29.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.763+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:29.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.763+0000 7f16f0da3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:29.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.763+0000 7f16f0da3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:29.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.763+0000 7f16f0da3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:29.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16eaffd640 1 -- 172.21.15.29:0/2821207078 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808701453 0 0) 0x7f16ec0a9b00 con 0x7f16ec13ead0 2023-08-22T23:22:29.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16eaffd640 10 monclient(hunting): _negotiate 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16eaffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16eaffd640 10 monclient(hunting): my global_id is 4264 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16eaffd640 1 -- 172.21.15.29:0/2821207078 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d4003520 con 0x7f16ec13ead0 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 1 -- 172.21.15.29:0/2821207078 >> v2:172.21.15.29:3300/0 conn(0x7f16ec0a7b90 msgr2=0x7f16ec13e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:29.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 1 --2- 172.21.15.29:0/2821207078 >> v2:172.21.15.29:3300/0 conn(0x7f16ec0a7b90 0x7f16ec13e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 1 -- 172.21.15.29:0/2821207078 >> v1:172.21.15.52:6789/0 conn(0x7f16ec13ead0 legacy=0x7f16ec1417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:29.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 10 monclient: _finish_hunting 0 2023-08-22T23:22:29.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 1 monclient: found mon.noname-b 2023-08-22T23:22:29.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:29.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:29.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.765+0000 7f16f0da3640 1 -- 172.21.15.29:0/2821207078 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec1412c0 con 0x7f16ec0b1380 2023-08-22T23:22:29.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.768+0000 7f16f0da3640 1 --2- 172.21.15.29:0/2821207078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0b1380 0x7f16ec14ba30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009d90 tx=0x7f16e402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a07a76cf8e353848 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:29.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 1 -- 172.21.15.29:0/2821207078 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e402cc50 con 0x7f16ec0b1380 2023-08-22T23:22:29.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 10 monclient: dump: 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16f2806640 5 monclient: authenticate success, global_id 4235 2023-08-22T23:22:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.771+0000 7f16f2806640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16eaffd640 1 -- 172.21.15.29:0/2821207078 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e402cdb0 con 0x7f16ec0b1380 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16eaffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16eaffd640 1 -- 172.21.15.29:0/2821207078 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4034640 con 0x7f16ec0b1380 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16f2806640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16f2806640 4 set_mon_vals no callback set 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.773+0000 7f16eaffd640 10 monclient: dump: 2023-08-22T23:22:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:29.781 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 10 monclient: shutdown 2023-08-22T23:22:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 -- 172.21.15.29:0/2821207078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0b1380 msgr2=0x7f16ec14ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 --2- 172.21.15.29:0/2821207078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0b1380 0x7f16ec14ba30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009d90 tx=0x7f16e402bea0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 -- 172.21.15.29:0/2821207078 shutdown_connections 2023-08-22T23:22:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 --2- 172.21.15.29:0/2821207078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0b1380 0x7f16ec14ba30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 --2- 172.21.15.29:0/2821207078 >> v2:172.21.15.29:3300/0 conn(0x7f16ec0a7b90 0x7f16ec13e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 -- 172.21.15.29:0/2821207078 >> 172.21.15.29:0/2821207078 conn(0x7f16ec0b6a00 msgr2=0x7f16ec0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 -- 172.21.15.29:0/2821207078 shutdown_connections 2023-08-22T23:22:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.774+0000 7f16f2806640 1 -- 172.21.15.29:0/2821207078 wait complete. 2023-08-22T23:22:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 10 monclient: build_initial_monmap 2023-08-22T23:22:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 10 monclient: monmap: 2023-08-22T23:22:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:29.787 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 1 librados: starting msgr at 2023-08-22T23:22:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 1 librados: starting objecter 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 1 Processor -- start 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 1 -- start start 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 1 librados: setting wanted keys 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 1 librados: calling monclient init 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.775+0000 7f16f2806640 10 monclient: init 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0a7b90 0x7f16ec13a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): picked mon.noname-c con 0x7f16ec0a7b90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16ec0b1380 0x7f16ec142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): picked mon.noname-a con 0x7f16ec0b1380 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 0x7f16ec1d9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): picked mon.noname-b con 0x7f16ec13ead0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec142bc0 con 0x7f16ec0b1380 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec13b320 con 0x7f16ec13ead0 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec143050 con 0x7f16ec0a7b90 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f2806640 10 monclient(hunting): authenticate will time out at 1514.025924s 2023-08-22T23:22:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16f0da3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 0x7f16ec1d9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.776+0000 7f16ebfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0a7b90 0x7f16ec13a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16eb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16ec0b1380 0x7f16ec142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 0x7f16ec1d9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39482/0 (socket says 172.21.15.29:39482) 2023-08-22T23:22:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 1 -- 172.21.15.29:0/1261597689 learned_addr learned my addr 172.21.15.29:0/1261597689 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): get_auth_request con 0x7f16ec13ead0 auth_method 0 2023-08-22T23:22:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16eb7fe640 10 monclient(hunting): get_auth_request con 0x7f16ec0b1380 auth_method 0 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16eb7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16ebfff640 10 monclient(hunting): get_auth_request con 0x7f16ec0a7b90 auth_method 0 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16ebfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.777+0000 7f16f0da3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.778+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.778+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.778+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.778+0000 7f16eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.778+0000 7f16eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.778+0000 7f16eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 10 monclient(hunting): handle_auth_done global_id 4267 payload 824 2023-08-22T23:22:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 1 -- 172.21.15.29:0/1261597689 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0a7b90 msgr2=0x7f16ec13a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0a7b90 0x7f16ec13a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 1 -- 172.21.15.29:0/1261597689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16ec0b1380 msgr2=0x7f16ec142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16ec0b1380 0x7f16ec142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 10 monclient: _finish_hunting 0 2023-08-22T23:22:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 1 monclient: found mon.noname-b 2023-08-22T23:22:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.780+0000 7f16f0da3640 1 -- 172.21.15.29:0/1261597689 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4009a70 con 0x7f16ec13ead0 2023-08-22T23:22:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.782+0000 7f16f0da3640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 0x7f16ec1d9100 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16e4037040 tx=0x7f16e4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.782+0000 7f16e8ff9640 10 client.?.objecter ms_handle_connect 0x7f16ec13ead0 2023-08-22T23:22:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.782+0000 7f16e8ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4004800 con 0x7f16ec13ead0 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: dump: 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e4009450 con 0x7f16ec13ead0 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4034680 con 0x7f16ec13ead0 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16e8ff9640 10 monclient: dump: 2023-08-22T23:22:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 5 monclient: authenticate success, global_id 4267 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 10 monclient: _renew_subs 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16ea7fc640 4 set_mon_vals no callback set 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec142760 con 0x7f16ec13ead0 2023-08-22T23:22:29.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 10 monclient: _renew_subs 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec1d9820 con 0x7f16ec13ead0 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.786+0000 7f16f2806640 1 librados: init done 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.787+0000 7f16f2806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.787+0000 7f16f2806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.787+0000 7f16f2806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:29.787+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec142d30 con 0x7f16ec13ead0 2023-08-22T23:22:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.054+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f16e40347e0 con 0x7f16ec13ead0 2023-08-22T23:22:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.055+0000 7f16e8ff9640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f16d806ba30 0x7f16d806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.055+0000 7f16ebfff640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f16d806ba30 0x7f16d806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.055+0000 7f16ebfff640 10 monclient: get_auth_request con 0x7f16d806ba30 auth_method 0 2023-08-22T23:22:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.056+0000 7f16ebfff640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f16d806ba30 0x7f16d806def0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009b40 tx=0x7f16dc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.056+0000 7f16e8ff9640 10 client.4267.objecter ms_handle_connect 0x7f16d806ba30 2023-08-22T23:22:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.060+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f16e407dd90 con 0x7f16ec13ead0 2023-08-22T23:22:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.060+0000 7f16e8ff9640 10 client.4267.objecter ms_dispatch 0x7f16ec000b90 osd_map(20..20 src has 1..20) v4 2023-08-22T23:22:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.060+0000 7f16e8ff9640 3 client.4267.objecter handle_osd_map got epochs [20,20] > 0 2023-08-22T23:22:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.060+0000 7f16e8ff9640 3 client.4267.objecter handle_osd_map decoding full epoch 20 2023-08-22T23:22:30.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.060+0000 7f16e8ff9640 20 client.4267.objecter dump_active .. 0 homeless 2023-08-22T23:22:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.078+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f16e40763c0 con 0x7f16ec13ead0 2023-08-22T23:22:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.078+0000 7f16e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.078+0000 7f16e8ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:30.169 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:15.452 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.264 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:15.547 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.282 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:15.579 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.317 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:15.600 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.336 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:15.633 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.340+0000 7f16f2806640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-08-22T23:22:30.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.340+0000 7f16f2806640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-08-22T23:22:30.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.340+0000 7f16f2806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:30.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.340+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f16ec04dbd0 con 0x7f16ec13ead0 2023-08-22T23:22:30.381 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:15.664 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.569+0000 7f16e8ff9640 1 -- 172.21.15.29:0/1261597689 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+169487 (secure 0 0 0) 0x7f16e4039090 con 0x7f16ec13ead0 2023-08-22T23:22:30.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.569+0000 7f16e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-08-22T23:22:30.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.569+0000 7f16e8ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:30.573 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:22:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 10 librados: watch_flush enter 2023-08-22T23:22:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 10 librados: watch_flush exit 2023-08-22T23:22:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 20 client.4267.objecter shutdown clearing up homeless session... 2023-08-22T23:22:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 10 client.4267.objecter successfully canceled tick 2023-08-22T23:22:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f16d806ba30 msgr2=0x7f16d806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f16d806ba30 0x7f16d806def0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009b40 tx=0x7f16dc009190 comp rx=0 tx=0).stop 2023-08-22T23:22:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 10 monclient: shutdown 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 msgr2=0x7f16ec1d9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 0x7f16ec1d9100 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16e4037040 tx=0x7f16e4004a40 comp rx=0 tx=0).stop 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 shutdown_connections 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f16d806ba30 0x7f16d806def0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16ec13ead0 0x7f16ec1d9100 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16ec0b1380 0x7f16ec142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 --2- 172.21.15.29:0/1261597689 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f16ec0a7b90 0x7f16ec13a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.571+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 >> 172.21.15.29:0/1261597689 conn(0x7f16ec0b6a00 msgr2=0x7f16ec13cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.572+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 shutdown_connections 2023-08-22T23:22:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.572+0000 7f16f2806640 1 -- 172.21.15.29:0/1261597689 wait complete. 2023-08-22T23:22:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.572+0000 7f16f2806640 1 librados: shutdown 2023-08-22T23:22:30.583 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":697719510},{"type":"v1","addr":"172.21.15.52:6801","nonce":697719510}]},"active_addr":"172.21.15.52:6801/697719510","active_change":"2023-08-22T23:21:37.942309+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":619219573}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2158740844}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":882760146}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3307837254}]}]} 2023-08-22T23:22:30.589 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-08-22T23:22:30.590 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-08-22T23:22:30.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-22T23:22:30.612 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:15.895 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.638 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:15.921 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.709+0000 7f8fe495c640 10 monclient: get_monmap_and_config 2023-08-22T23:22:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.709+0000 7f8fe495c640 10 monclient: build_initial_monmap 2023-08-22T23:22:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.709+0000 7f8fe495c640 10 monclient: monmap: 2023-08-22T23:22:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:30.710564+0000 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:30.710564+0000 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:30.713 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 Processor -- start 2023-08-22T23:22:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 -- start start 2023-08-22T23:22:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient: init 2023-08-22T23:22:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fe00a8210 0x7f8fe00a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe00a8210 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fe00b01e0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f8fe00b0600 0x7f8fe0144da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe00b0600 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00a7770 con 0x7f8fe00b01e0 2023-08-22T23:22:30.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:30.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe01452e0 con 0x7f8fe00a8210 2023-08-22T23:22:30.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:30.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe00a8b30 con 0x7f8fe00b0600 2023-08-22T23:22:30.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:30.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.710+0000 7f8fe495c640 10 monclient(hunting): authenticate will time out at 1514.961371s 2023-08-22T23:22:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.755+0000 7f8fde59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fe00a8210 0x7f8fe00a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.757+0000 7f8fd7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8fe00b01e0 0x7f8fe0144690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59894/0 (socket says 172.21.15.29:59894) 2023-08-22T23:22:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.757+0000 7f8fd7fff640 1 -- 172.21.15.29:0/257377712 learned_addr learned my addr 172.21.15.29:0/257377712 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.757+0000 7f8fde59c640 10 monclient(hunting): get_auth_request con 0x7f8fe00a8210 auth_method 0 2023-08-22T23:22:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.757+0000 7f8fde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.757+0000 7f8fde59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.757+0000 7f8fde59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.761+0000 7f8fde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.761+0000 7f8fde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.761+0000 7f8fde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-08-22T23:22:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 1 -- 172.21.15.29:0/257377712 >> v2:172.21.15.52:3300/0 conn(0x7f8fe00b0600 msgr2=0x7f8fe0144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 1 --2- 172.21.15.29:0/257377712 >> v2:172.21.15.52:3300/0 conn(0x7f8fe00b0600 0x7f8fe0144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 1 -- 172.21.15.29:0/257377712 >> v1:172.21.15.29:6789/0 conn(0x7f8fe00b01e0 legacy=0x7f8fe0144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 10 monclient: _finish_hunting 0 2023-08-22T23:22:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 1 monclient: found mon.noname-a 2023-08-22T23:22:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.764+0000 7f8fde59c640 1 -- 172.21.15.29:0/257377712 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00a8d30 con 0x7f8fe00a8210 2023-08-22T23:22:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.789+0000 7f8fde59c640 1 --2- 172.21.15.29:0/257377712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 0x7f8fe00a85f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00b2b20 tx=0x7f8fd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab355e43cce94d8 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:30.793 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:16.090 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 1 -- 172.21.15.29:0/257377712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0031830 con 0x7f8fe00a8210 2023-08-22T23:22:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: dump: 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: _finish_auth 0 2023-08-22T23:22:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: _check_auth_tickets 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 1 -- 172.21.15.29:0/257377712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd0031990 con 0x7f8fe00a8210 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fe495c640 5 monclient: authenticate success, global_id 4272 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fe495c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 1 -- 172.21.15.29:0/257377712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0031d20 con 0x7f8fe00a8210 2023-08-22T23:22:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fe495c640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fe495c640 4 set_mon_vals no callback set 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.842+0000 7f8fddd9b640 10 monclient: dump: 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:30.849 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 10 monclient: shutdown 2023-08-22T23:22:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 -- 172.21.15.29:0/257377712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 msgr2=0x7f8fe00a85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 --2- 172.21.15.29:0/257377712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 0x7f8fe00a85f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00b2b20 tx=0x7f8fd002b8d0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 -- 172.21.15.29:0/257377712 shutdown_connections 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 --2- 172.21.15.29:0/257377712 >> v2:172.21.15.52:3300/0 conn(0x7f8fe00b0600 0x7f8fe0144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 --2- 172.21.15.29:0/257377712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 0x7f8fe00a85f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 -- 172.21.15.29:0/257377712 >> 172.21.15.29:0/257377712 conn(0x7f8fe00b6a10 msgr2=0x7f8fe00b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 -- 172.21.15.29:0/257377712 shutdown_connections 2023-08-22T23:22:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.843+0000 7f8fe495c640 1 -- 172.21.15.29:0/257377712 wait complete. 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.844+0000 7f8fe495c640 10 monclient: build_initial_monmap 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.844+0000 7f8fe495c640 10 monclient: monmap: 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 librados: starting msgr at 2023-08-22T23:22:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 librados: starting objecter 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 Processor -- start 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 -- start start 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 librados: setting wanted keys 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 librados: calling monclient init 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 10 monclient: init 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.845+0000 7f8fe495c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 0x7f8fe0138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe00a8210 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 0x7f8fe01d9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe00b01e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fe00b0600 0x7f8fe01d99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe00b0600 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe01421d0 con 0x7f8fe00a8210 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0142490 con 0x7f8fe00b01e0 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0144770 con 0x7f8fe00b0600 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.846+0000 7f8fe495c640 10 monclient(hunting): authenticate will time out at 1515.096546s 2023-08-22T23:22:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.860+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 0x7f8fe01d9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.862+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 0x7f8fe01d9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39500/0 (socket says 172.21.15.29:39500) 2023-08-22T23:22:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.862+0000 7f8fd7fff640 1 -- 172.21.15.29:0/383779563 learned_addr learned my addr 172.21.15.29:0/383779563 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.862+0000 7f8fd7fff640 10 monclient(hunting): get_auth_request con 0x7f8fe00b01e0 auth_method 0 2023-08-22T23:22:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.862+0000 7f8fd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.862+0000 7f8fd7fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.862+0000 7f8fd7fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.863+0000 7f8fd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.863+0000 7f8fd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.863+0000 7f8fd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 1 -- 172.21.15.29:0/383779563 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fe00b0600 msgr2=0x7f8fe01d99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fe00b0600 0x7f8fe01d99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 1 -- 172.21.15.29:0/383779563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 msgr2=0x7f8fe0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 0x7f8fe0138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 1 monclient: found mon.noname-b 2023-08-22T23:22:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.867+0000 7f8fd7fff640 1 -- 172.21.15.29:0/383779563 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd0009400 con 0x7f8fe00b01e0 2023-08-22T23:22:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.884+0000 7f8fd7fff640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 0x7f8fe01d9460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c400 tx=0x7f8fc800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.884+0000 7f8fd6ffd640 10 client.?.objecter ms_handle_connect 0x7f8fe00b01e0 2023-08-22T23:22:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.884+0000 7f8fd6ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:30.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8005ce0 con 0x7f8fe00b01e0 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: dump: 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.939 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:30.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc8005e40 con 0x7f8fe00b01e0 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80072d0 con 0x7f8fe00b01e0 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 5 monclient: authenticate success, global_id 4270 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 10 monclient: _renew_subs 2023-08-22T23:22:30.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe013d5f0 con 0x7f8fe00b01e0 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fdd59a640 4 set_mon_vals no callback set 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fd6ffd640 10 monclient: dump: 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:30.942 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:30.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 10 monclient: _renew_subs 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe0142d30 con 0x7f8fe00b01e0 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.934+0000 7f8fe495c640 1 librados: init done 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.935+0000 7f8fe495c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.935+0000 7f8fe495c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:30.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.935+0000 7f8fe495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:30.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.935+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0066cb0 con 0x7f8fe00b01e0 2023-08-22T23:22:30.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.958+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8fc80074f0 con 0x7f8fe00b01e0 2023-08-22T23:22:30.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.958+0000 7f8fd6ffd640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8fc006ba30 0x7f8fc006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:30.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.959+0000 7f8fde59c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8fc006ba30 0x7f8fc006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:30.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.959+0000 7f8fde59c640 10 monclient: get_auth_request con 0x7f8fc006ba30 auth_method 0 2023-08-22T23:22:30.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.959+0000 7f8fde59c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8fc006ba30 0x7f8fc006def0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0036000 tx=0x7f8fd0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:30.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:30.959+0000 7f8fd6ffd640 10 client.4270.objecter ms_handle_connect 0x7f8fc006ba30 2023-08-22T23:22:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.227+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8fc8057ab0 con 0x7f8fe00b01e0 2023-08-22T23:22:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.227+0000 7f8fd6ffd640 10 client.4270.objecter ms_dispatch 0x7f8fe0000b90 osd_map(20..20 src has 1..20) v4 2023-08-22T23:22:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.227+0000 7f8fd6ffd640 3 client.4270.objecter handle_osd_map got epochs [20,20] > 0 2023-08-22T23:22:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.227+0000 7f8fd6ffd640 3 client.4270.objecter handle_osd_map decoding full epoch 20 2023-08-22T23:22:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.228+0000 7f8fd6ffd640 20 client.4270.objecter dump_active .. 0 homeless 2023-08-22T23:22:31.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.279+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f8fc8053760 con 0x7f8fe00b01e0 2023-08-22T23:22:31.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.279+0000 7f8fd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:31.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.279+0000 7f8fd6ffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:31.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.460+0000 7f8fe495c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:31.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.460+0000 7f8fe495c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:31.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.460+0000 7f8fe495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.460+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fe006a9d0 con 0x7f8fe00b01e0 2023-08-22T23:22:31.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.540+0000 7f8fd6ffd640 1 -- 172.21.15.29:0/383779563 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14639 (secure 0 0 0) 0x7f8fc8017090 con 0x7f8fe00b01e0 2023-08-22T23:22:31.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.540+0000 7f8fd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:31.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.540+0000 7f8fd6ffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:22:31.543 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:22:30.599904+0000","last_up_change":"2023-08-22T23:22:22.559235+0000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-22T23:22:20.507839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-22T23:22:27.474738+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6809","nonce":1473840652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6811","nonce":1473840652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6816","nonce":1473840652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6813","nonce":1473840652}]},"public_addr":"172.21.15.29:6809/1473840652","cluster_addr":"172.21.15.29:6811/1473840652","heartbeat_back_addr":"172.21.15.29:6816/1473840652","heartbeat_front_addr":"172.21.15.29:6813/1473840652","state":["exists","up"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6825","nonce":2619884743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6827","nonce":2619884743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6831","nonce":2619884743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6829","nonce":2619884743}]},"public_addr":"172.21.15.29:6825/2619884743","cluster_addr":"172.21.15.29:6827/2619884743","heartbeat_back_addr":"172.21.15.29:6831/2619884743","heartbeat_front_addr":"172.21.15.29:6829/2619884743","state":["exists","up"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6801","nonce":1029571786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6803","nonce":1029571786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6807","nonce":1029571786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6805","nonce":1029571786}]},"public_addr":"172.21.15.29:6801/1029571786","cluster_addr":"172.21.15.29:6803/1029571786","heartbeat_back_addr":"172.21.15.29:6807/1029571786","heartbeat_front_addr":"172.21.15.29:6805/1029571786","state":["exists","up"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6817","nonce":4227673673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6819","nonce":4227673673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6823","nonce":4227673673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6821","nonce":4227673673}]},"public_addr":"172.21.15.29:6817/4227673673","cluster_addr":"172.21.15.29:6819/4227673673","heartbeat_back_addr":"172.21.15.29:6823/4227673673","heartbeat_front_addr":"172.21.15.29:6821/4227673673","state":["exists","up"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6803","nonce":2157117708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6805","nonce":2157117708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6809","nonce":2157117708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6807","nonce":2157117708}]},"public_addr":"172.21.15.52:6803/2157117708","cluster_addr":"172.21.15.52:6805/2157117708","heartbeat_back_addr":"172.21.15.52:6809/2157117708","heartbeat_front_addr":"172.21.15.52:6807/2157117708","state":["exists","up"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6827","nonce":4164832567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6829","nonce":4164832567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6833","nonce":4164832567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6831","nonce":4164832567}]},"public_addr":"172.21.15.52:6827/4164832567","cluster_addr":"172.21.15.52:6829/4164832567","heartbeat_back_addr":"172.21.15.52:6833/4164832567","heartbeat_front_addr":"172.21.15.52:6831/4164832567","state":["exists","up"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6819","nonce":2556931765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6821","nonce":2556931765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6825","nonce":2556931765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6823","nonce":2556931765}]},"public_addr":"172.21.15.52:6819/2556931765","cluster_addr":"172.21.15.52:6821/2556931765","heartbeat_back_addr":"172.21.15.52:6825/2556931765","heartbeat_front_addr":"172.21.15.52:6823/2556931765","state":["exists","up"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6811","nonce":3454580439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6813","nonce":3454580439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6817","nonce":3454580439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6815","nonce":3454580439}]},"public_addr":"172.21.15.52:6811/3454580439","cluster_addr":"172.21.15.52:6813/3454580439","heartbeat_back_addr":"172.21.15.52:6817/3454580439","heartbeat_front_addr":"172.21.15.52:6815/3454580439","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:19.045553+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.775383+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.732376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.515362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.723819+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.012554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.629140+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.069717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:22:31.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 10 librados: watch_flush enter 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 10 librados: watch_flush exit 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 20 client.4270.objecter shutdown clearing up homeless session... 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 10 client.4270.objecter successfully canceled tick 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8fc006ba30 msgr2=0x7f8fc006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8fc006ba30 0x7f8fc006def0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0036000 tx=0x7f8fd0030000 comp rx=0 tx=0).stop 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 10 monclient: shutdown 2023-08-22T23:22:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 msgr2=0x7f8fe01d9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 0x7f8fe01d9460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c400 tx=0x7f8fc800c8c0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 shutdown_connections 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8fc006ba30 0x7f8fc006def0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8fe00b0600 0x7f8fe01d99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8fe00b01e0 0x7f8fe01d9460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.542+0000 7f8fe495c640 1 --2- 172.21.15.29:0/383779563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fe00a8210 0x7f8fe0138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.543+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 >> 172.21.15.29:0/383779563 conn(0x7f8fe00b6a10 msgr2=0x7f8fe00b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:31.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.543+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 shutdown_connections 2023-08-22T23:22:31.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.543+0000 7f8fe495c640 1 -- 172.21.15.29:0/383779563 wait complete. 2023-08-22T23:22:31.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.543+0000 7f8fe495c640 1 librados: shutdown 2023-08-22T23:22:31.555 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-08-22T23:22:31.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-22T23:22:31.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.705+0000 7f4f4d863640 10 monclient: get_monmap_and_config 2023-08-22T23:22:31.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.705+0000 7f4f4d863640 10 monclient: build_initial_monmap 2023-08-22T23:22:31.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.705+0000 7f4f4d863640 10 monclient: monmap: 2023-08-22T23:22:31.707 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:31.707 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:31.707 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:31.706822+0000 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:31.706822+0000 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:31.708 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:31.709 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:31.709 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:31.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 Processor -- start 2023-08-22T23:22:31.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 -- start start 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient: init 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f480a7b90 0x7f4f4813e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): picked mon.noname-a con 0x7f4f480a7b90 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f4813ead0 0x7f4f481417e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4813ead0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4f480b1380 0x7f4f481446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480b1380 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f480a9b00 con 0x7f4f480a7b90 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f480a9de0 con 0x7f4f480b1380 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480aa0c0 con 0x7f4f4813ead0 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.706+0000 7f4f4d863640 10 monclient(hunting): authenticate will time out at 1515.957395s 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.707+0000 7f4f46ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f480a7b90 0x7f4f4813e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.707+0000 7f4f477fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4f480b1380 0x7f4f481446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.707+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f4813ead0 0x7f4f481417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.707+0000 7f4f46ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f480a7b90 0x7f4f4813e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33294/0 (socket says 172.21.15.29:33294) 2023-08-22T23:22:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.707+0000 7f4f46ffd640 1 -- 172.21.15.29:0/545839974 learned_addr learned my addr 172.21.15.29:0/545839974 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.707+0000 7f4f46ffd640 10 monclient(hunting): get_auth_request con 0x7f4f480a7b90 auth_method 0 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f46ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f46ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): get_auth_request con 0x7f4f480b1380 auth_method 0 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): get_auth_request con 0x7f4f4813ead0 auth_method 0 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:31.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:31.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:31.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:31.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:31.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.708+0000 7f4f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:31.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2023-08-22T23:22:31.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 1 -- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f4813ead0 msgr2=0x7f4f481417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:31.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 1 --2- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f4813ead0 0x7f4f481417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 1 -- 172.21.15.29:0/545839974 >> v2:172.21.15.52:3300/0 conn(0x7f4f480b1380 msgr2=0x7f4f481446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:31.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 1 --2- 172.21.15.29:0/545839974 >> v2:172.21.15.52:3300/0 conn(0x7f4f480b1380 0x7f4f481446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 1 monclient: found mon.noname-a 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f46ffd640 1 -- 172.21.15.29:0/545839974 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f481412c0 con 0x7f4f480a7b90 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.710+0000 7f4f3ffff640 1 --2- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f4813ead0 0x7f4f481417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:22:31.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.712+0000 7f4f46ffd640 1 --2- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f480a7b90 0x7f4f4813e590 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f480b2b10 tx=0x7f4f3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c2b43de4853ecc server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:31.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 1 -- 172.21.15.29:0/545839974 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38031870 con 0x7f4f480a7b90 2023-08-22T23:22:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: dump: 2023-08-22T23:22:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 1 -- 172.21.15.29:0/545839974 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f380319d0 con 0x7f4f480a7b90 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 1 -- 172.21.15.29:0/545839974 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38031d60 con 0x7f4f480a7b90 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:31.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f467fc640 10 monclient: dump: 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:31.723 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f4d863640 5 monclient: authenticate success, global_id 4278 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f4d863640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.716+0000 7f4f4d863640 4 set_mon_vals no callback set 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 10 monclient: shutdown 2023-08-22T23:22:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 -- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f480a7b90 msgr2=0x7f4f4813e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 --2- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f480a7b90 0x7f4f4813e590 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f480b2b10 tx=0x7f4f3802b910 comp rx=0 tx=0).stop 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 -- 172.21.15.29:0/545839974 shutdown_connections 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 --2- 172.21.15.29:0/545839974 >> v2:172.21.15.52:3300/0 conn(0x7f4f480b1380 0x7f4f481446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 --2- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f4813ead0 0x7f4f481417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 --2- 172.21.15.29:0/545839974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f480a7b90 0x7f4f4813e590 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 -- 172.21.15.29:0/545839974 >> 172.21.15.29:0/545839974 conn(0x7f4f480b6a00 msgr2=0x7f4f480b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 -- 172.21.15.29:0/545839974 shutdown_connections 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.717+0000 7f4f4d863640 1 -- 172.21.15.29:0/545839974 wait complete. 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 10 monclient: build_initial_monmap 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 10 monclient: monmap: 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 1 librados: starting msgr at 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 1 librados: starting objecter 2023-08-22T23:22:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 1 Processor -- start 2023-08-22T23:22:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 1 -- start start 2023-08-22T23:22:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 1 librados: setting wanted keys 2023-08-22T23:22:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 1 librados: calling monclient init 2023-08-22T23:22:31.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.718+0000 7f4f4d863640 10 monclient: init 2023-08-22T23:22:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:22:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f480a7b90 0x7f4f480b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a7b90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f480b1380 0x7f4f481d96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): picked mon.noname-b con 0x7f4f480b1380 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 0x7f4f480af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4813ead0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f481d9da0 con 0x7f4f4813ead0 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480b66f0 con 0x7f4f480b1380 2023-08-22T23:22:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480b0140 con 0x7f4f480a7b90 2023-08-22T23:22:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f4d863640 10 monclient(hunting): authenticate will time out at 1515.969803s 2023-08-22T23:22:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 0x7f4f480af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f46ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f480a7b90 0x7f4f480b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.719+0000 7f4f3ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f480b1380 0x7f4f481d96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 0x7f4f480af5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33308/0 (socket says 172.21.15.29:33308) 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 1 -- 172.21.15.29:0/3432386195 learned_addr learned my addr 172.21.15.29:0/3432386195 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): get_auth_request con 0x7f4f4813ead0 auth_method 0 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:31.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): get_auth_request con 0x7f4f480b1380 auth_method 0 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): get_auth_request con 0x7f4f480a7b90 auth_method 0 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:31.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:31.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.720+0000 7f4f46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 1 -- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f480a7b90 msgr2=0x7f4f480b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f480a7b90 0x7f4f480b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 1 -- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f480b1380 msgr2=0x7f4f481d96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f480b1380 0x7f4f481d96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 1 monclient: found mon.noname-a 2023-08-22T23:22:31.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:31.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:31.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.723+0000 7f4f477fe640 1 -- 172.21.15.29:0/3432386195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009400 con 0x7f4f4813ead0 2023-08-22T23:22:31.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.726+0000 7f4f477fe640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 0x7f4f480af5c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000eea0 tx=0x7f4f4000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:31.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.726+0000 7f4f44ff9640 10 client.?.objecter ms_handle_connect 0x7f4f4813ead0 2023-08-22T23:22:31.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.726+0000 7f4f44ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:31.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40010040 con 0x7f4f4813ead0 2023-08-22T23:22:31.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:31.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:31.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: dump: 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:31.739 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f40014400 con 0x7f4f4813ead0 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:31.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f400147b0 con 0x7f4f4813ead0 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f44ff9640 10 monclient: dump: 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:31.741 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 5 monclient: authenticate success, global_id 4281 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 10 monclient: _renew_subs 2023-08-22T23:22:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f380094d0 con 0x7f4f4813ead0 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f45ffb640 4 set_mon_vals no callback set 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 10 monclient: _renew_subs 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f480afce0 con 0x7f4f4813ead0 2023-08-22T23:22:31.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.729+0000 7f4f4d863640 1 librados: init done 2023-08-22T23:22:31.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.730+0000 7f4f4d863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:31.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.731+0000 7f4f4d863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:31.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.731+0000 7f4f4d863640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:31.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.731+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f480aff60 con 0x7f4f4813ead0 2023-08-22T23:22:31.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.829+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4f40014910 con 0x7f4f4813ead0 2023-08-22T23:22:31.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.829+0000 7f4f44ff9640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f4f2806ba10 0x7f4f2806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:31.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.830+0000 7f4f46ffd640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f4f2806ba10 0x7f4f2806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:31.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.830+0000 7f4f46ffd640 10 monclient: get_auth_request con 0x7f4f2806ba10 auth_method 0 2023-08-22T23:22:31.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.830+0000 7f4f44ff9640 10 client.4281.objecter ms_handle_connect 0x7f4f2806ba10 2023-08-22T23:22:31.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:31.830+0000 7f4f46ffd640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f4f2806ba10 0x7f4f2806ded0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f38036000 tx=0x7f4f3803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4f4005c5d0 con 0x7f4f4813ead0 2023-08-22T23:22:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 10 client.4281.objecter ms_dispatch 0x7f4f48000b90 osd_map(21..21 src has 1..21) v4 2023-08-22T23:22:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 3 client.4281.objecter handle_osd_map got epochs [21,21] > 0 2023-08-22T23:22:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 3 client.4281.objecter handle_osd_map decoding full epoch 21 2023-08-22T23:22:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 20 client.4281.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f4f40054c00 con 0x7f4f4813ead0 2023-08-22T23:22:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.009+0000 7f4f44ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.207+0000 7f4f4d863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.207+0000 7f4f4d863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.207+0000 7f4f4d863640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.207+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f480b03d0 con 0x7f4f4813ead0 2023-08-22T23:22:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.330+0000 7f4f44ff9640 1 -- 172.21.15.29:0/3432386195 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14626 (secure 0 0 0) 0x7f4f40018090 con 0x7f4f4813ead0 2023-08-22T23:22:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.330+0000 7f4f44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:22:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.330+0000 7f4f44ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:32.332 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:22:32.333 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:22:32.018236+0000","last_up_change":"2023-08-22T23:22:22.559235+0000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-22T23:22:20.507839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-22T23:22:27.474738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6809","nonce":1473840652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6811","nonce":1473840652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6816","nonce":1473840652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6813","nonce":1473840652}]},"public_addr":"172.21.15.29:6809/1473840652","cluster_addr":"172.21.15.29:6811/1473840652","heartbeat_back_addr":"172.21.15.29:6816/1473840652","heartbeat_front_addr":"172.21.15.29:6813/1473840652","state":["exists","up"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6825","nonce":2619884743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6827","nonce":2619884743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6831","nonce":2619884743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6829","nonce":2619884743}]},"public_addr":"172.21.15.29:6825/2619884743","cluster_addr":"172.21.15.29:6827/2619884743","heartbeat_back_addr":"172.21.15.29:6831/2619884743","heartbeat_front_addr":"172.21.15.29:6829/2619884743","state":["exists","up"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6801","nonce":1029571786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6803","nonce":1029571786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6807","nonce":1029571786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6805","nonce":1029571786}]},"public_addr":"172.21.15.29:6801/1029571786","cluster_addr":"172.21.15.29:6803/1029571786","heartbeat_back_addr":"172.21.15.29:6807/1029571786","heartbeat_front_addr":"172.21.15.29:6805/1029571786","state":["exists","up"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6817","nonce":4227673673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6819","nonce":4227673673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6823","nonce":4227673673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6821","nonce":4227673673}]},"public_addr":"172.21.15.29:6817/4227673673","cluster_addr":"172.21.15.29:6819/4227673673","heartbeat_back_addr":"172.21.15.29:6823/4227673673","heartbeat_front_addr":"172.21.15.29:6821/4227673673","state":["exists","up"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6803","nonce":2157117708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6805","nonce":2157117708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6809","nonce":2157117708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6807","nonce":2157117708}]},"public_addr":"172.21.15.52:6803/2157117708","cluster_addr":"172.21.15.52:6805/2157117708","heartbeat_back_addr":"172.21.15.52:6809/2157117708","heartbeat_front_addr":"172.21.15.52:6807/2157117708","state":["exists","up"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6827","nonce":4164832567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6829","nonce":4164832567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6833","nonce":4164832567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6831","nonce":4164832567}]},"public_addr":"172.21.15.52:6827/4164832567","cluster_addr":"172.21.15.52:6829/4164832567","heartbeat_back_addr":"172.21.15.52:6833/4164832567","heartbeat_front_addr":"172.21.15.52:6831/4164832567","state":["exists","up"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6819","nonce":2556931765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6821","nonce":2556931765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6825","nonce":2556931765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6823","nonce":2556931765}]},"public_addr":"172.21.15.52:6819/2556931765","cluster_addr":"172.21.15.52:6821/2556931765","heartbeat_back_addr":"172.21.15.52:6825/2556931765","heartbeat_front_addr":"172.21.15.52:6823/2556931765","state":["exists","up"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6811","nonce":3454580439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6813","nonce":3454580439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6817","nonce":3454580439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6815","nonce":3454580439}]},"public_addr":"172.21.15.52:6811/3454580439","cluster_addr":"172.21.15.52:6813/3454580439","heartbeat_back_addr":"172.21.15.52:6817/3454580439","heartbeat_front_addr":"172.21.15.52:6815/3454580439","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:19.045553+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.775383+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.732376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.515362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.723819+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.012554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.629140+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.069717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:22:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 10 librados: watch_flush enter 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 10 librados: watch_flush exit 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 20 client.4281.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 10 client.4281.objecter successfully canceled tick 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f4f2806ba10 msgr2=0x7f4f2806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f4f2806ba10 0x7f4f2806ded0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f38036000 tx=0x7f4f3803c040 comp rx=0 tx=0).stop 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 10 monclient: shutdown 2023-08-22T23:22:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 msgr2=0x7f4f480af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 0x7f4f480af5c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000eea0 tx=0x7f4f4000ef90 comp rx=0 tx=0).stop 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 shutdown_connections 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f4f2806ba10 0x7f4f2806ded0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f4813ead0 0x7f4f480af5c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f480b1380 0x7f4f481d96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 --2- 172.21.15.29:0/3432386195 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f4f480a7b90 0x7f4f480b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 >> 172.21.15.29:0/3432386195 conn(0x7f4f480b6a00 msgr2=0x7f4f481382f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 shutdown_connections 2023-08-22T23:22:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 -- 172.21.15.29:0/3432386195 wait complete. 2023-08-22T23:22:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.333+0000 7f4f4d863640 1 librados: shutdown 2023-08-22T23:22:32.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-08-22T23:22:32.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-08-22T23:22:32.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-08-22T23:22:32.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-08-22T23:22:32.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-08-22T23:22:32.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-08-22T23:22:32.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-08-22T23:22:32.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-08-22T23:22:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.531+0000 7fae43477640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.531+0000 7fae43477640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.531+0000 7fae43477640 10 monclient: monmap: 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.533099+0000 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.533099+0000 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.535 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7f0bdc8aa640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7f0bdc8aa640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7f0bdc8aa640 10 monclient: monmap: 2023-08-22T23:22:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.533586+0000 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.533586+0000 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.538 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7fae43477640 1 Processor -- start 2023-08-22T23:22:32.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7ffb7135d640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7ffb7135d640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7ffb7135d640 10 monclient: monmap: 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.533385+0000 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.533385+0000 2023-08-22T23:22:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.540 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7ffb7135d640 1 Processor -- start 2023-08-22T23:22:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7ffb7135d640 1 -- start start 2023-08-22T23:22:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7ffb7135d640 10 monclient: init 2023-08-22T23:22:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7f7802336640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7f7802336640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.532+0000 7f7802336640 10 monclient: monmap: 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.534149+0000 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.534149+0000 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.543 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f0bdc8aa640 1 Processor -- start 2023-08-22T23:22:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f42fe6c5640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f42fe6c5640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f42fe6c5640 10 monclient: monmap: 2023-08-22T23:22:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.534981+0000 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.534981+0000 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.546 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.546 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f9cb2857640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f9cb2857640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7fae43477640 1 -- start start 2023-08-22T23:22:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7fae43477640 10 monclient: init 2023-08-22T23:22:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-08-22T23:22:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fae3c0a7f80 0x7fae3c0a8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): picked mon.noname-c con 0x7fae3c0a7f80 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae3c0a7a10 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0b0920 0x7fae3c140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): picked mon.noname-b con 0x7fae3c0b0920 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0a9c00 con 0x7fae3c0a7a10 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0b0420 con 0x7fae3c0a7f80 2023-08-22T23:22:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0b0560 con 0x7fae3c0b0920 2023-08-22T23:22:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7fae43477640 10 monclient(hunting): authenticate will time out at 1516.784885s 2023-08-22T23:22:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f6ac997f640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f6ac997f640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f6ac997f640 10 monclient: monmap: 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.534658+0000 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.534658+0000 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.551 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.551 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.551 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.551 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f9cb2857640 10 monclient: monmap: 2023-08-22T23:22:32.551 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.535030+0000 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.535030+0000 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.552 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.553 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f0bdc8aa640 1 -- start start 2023-08-22T23:22:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7f0bdc8aa640 10 monclient: init 2023-08-22T23:22:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7f6ac997f640 1 Processor -- start 2023-08-22T23:22:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7f7802336640 1 Processor -- start 2023-08-22T23:22:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7ff494396640 10 monclient: get_monmap_and_config 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7ff494396640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.533+0000 7ff494396640 10 monclient: monmap: 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:32.535227+0000 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:32.535227+0000 2023-08-22T23:22:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:32.556 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-08-22T23:22:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bd80ae150 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0bd8142d60 0x7f0bd80712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd8142d60 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bd81457c0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd80717e0 con 0x7f0bd81457c0 2023-08-22T23:22:32.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd80acb90 con 0x7f0bd80ae150 2023-08-22T23:22:32.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd81431a0 con 0x7f0bd8142d60 2023-08-22T23:22:32.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f0bdc8aa640 10 monclient(hunting): authenticate will time out at 1516.785659s 2023-08-22T23:22:32.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ff494396640 1 Processor -- start 2023-08-22T23:22:32.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ff494396640 1 -- start start 2023-08-22T23:22:32.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ff494396640 10 monclient: init 2023-08-22T23:22:32.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0aa1a0 0x7ffb6c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6c0aa1a0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb6c0aafd0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb6c0adf30 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0a91f0 con 0x7ffb6c0aafd0 2023-08-22T23:22:32.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0abe30 con 0x7ffb6c0adf30 2023-08-22T23:22:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0aa580 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.534+0000 7ffb7135d640 10 monclient(hunting): authenticate will time out at 1516.784693s 2023-08-22T23:22:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7ffb6b7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7ffb6c0adf30 0x7ffb6c1500d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44542/0 (socket says 172.21.15.29:44542) 2023-08-22T23:22:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7ffb6b7fe640 1 -- 172.21.15.29:0/4100665875 learned_addr learned my addr 172.21.15.29:0/4100665875 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fae3c0a7f80 0x7fae3c0a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42c76640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0b0920 0x7fae3c140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f7802336640 1 -- start start 2023-08-22T23:22:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f7802336640 10 monclient: init 2023-08-22T23:22:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f6ac997f640 1 -- start start 2023-08-22T23:22:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f6ac997f640 10 monclient: init 2023-08-22T23:22:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fae3c0a7f80 0x7fae3c0a8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39518/0 (socket says 172.21.15.29:39518) 2023-08-22T23:22:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 1 -- 172.21.15.29:0/935910049 learned_addr learned my addr 172.21.15.29:0/935910049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 10 monclient(hunting): get_auth_request con 0x7fae3c0a7f80 auth_method 0 2023-08-22T23:22:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f42fe6c5640 1 Processor -- start 2023-08-22T23:22:32.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f42fe6c5640 1 -- start start 2023-08-22T23:22:32.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7f42fe6c5640 10 monclient: init 2023-08-22T23:22:32.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.535+0000 7fae42475640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f9cb2857640 1 Processor -- start 2023-08-22T23:22:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-08-22T23:22:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ac40a9e70 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 0x7f6ac40a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac40b2df0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ac40abc70 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40a9950 con 0x7f6ac40abc70 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40ab060 con 0x7f6ac40a9e70 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40aa310 con 0x7f6ac40b2df0 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac997f640 10 monclient(hunting): authenticate will time out at 1516.786703s 2023-08-22T23:22:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-08-22T23:22:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f77f4034aa0 0x7f77f402a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): picked mon.noname-c con 0x7f77f4034aa0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77f4034e80 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77f40c3d00 0x7f77f40c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): picked mon.noname-a con 0x7f77f40c3d00 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f4032f60 con 0x7f77f4034e80 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f4033850 con 0x7f77f40c3d00 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f402cb10 con 0x7f77f4034aa0 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f7802336640 10 monclient(hunting): authenticate will time out at 1516.786829s 2023-08-22T23:22:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f0bd659c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f0bd80ae150 0x7f0bd8142650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44554/0 (socket says 172.21.15.29:44554) 2023-08-22T23:22:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f0bd659c640 1 -- 172.21.15.29:0/962841963 learned_addr learned my addr 172.21.15.29:0/962841963 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f9cb2857640 1 -- start start 2023-08-22T23:22:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f9cb2857640 10 monclient: init 2023-08-22T23:22:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-08-22T23:22:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff48c0aa1a0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ff48c0aafd0 0x7ff48c0adf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient(hunting): picked mon.noname-c con 0x7ff48c0aafd0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff48c149380 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c0a91f0 con 0x7ff48c0aa1a0 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c0abe30 con 0x7ff48c149380 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48c0aa580 con 0x7ff48c0aafd0 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ff494396640 10 monclient(hunting): authenticate will time out at 1516.786733s 2023-08-22T23:22:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7fae42c76640 10 monclient(hunting): get_auth_request con 0x7fae3c0b0920 auth_method 0 2023-08-22T23:22:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7fae42c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7fae42c76640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7fae42c76640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7ffb6affd640 1 --2- 172.21.15.29:0/4100665875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0aa1a0 0x7ffb6c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f42fe6c5640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f42fe6c5640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-08-22T23:22:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f42fe6c5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f8071750 0x7f42f80ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f42fe6c5640 10 monclient(hunting): picked mon.noname-b con 0x7f42f8071750 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f42fe6c5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f80aa510 0x7f42f80a9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f42fe6c5640 10 monclient(hunting): picked mon.noname-a con 0x7f42f80aa510 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:32.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.536+0000 7f6ac3fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 0x7f6ac40a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42f80a8900 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f80aaa30 con 0x7f42f80a8900 2023-08-22T23:22:32.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f8071b30 con 0x7f42f80aa510 2023-08-22T23:22:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80ac820 con 0x7f42f8071750 2023-08-22T23:22:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f42fe6c5640 10 monclient(hunting): authenticate will time out at 1516.787640s 2023-08-22T23:22:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7fae42475640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7fae42475640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7fae42475640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f6ac3fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 0x7f6ac40a86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43334/0 (socket says 172.21.15.29:43334) 2023-08-22T23:22:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f6ac3fff640 1 -- 172.21.15.29:0/3997040306 learned_addr learned my addr 172.21.15.29:0/3997040306 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f6ac3fff640 10 monclient(hunting): get_auth_request con 0x7f6ac40b2df0 auth_method 0 2023-08-22T23:22:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f6ac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f6ac3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f6ac3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cac0adfc0 0x7f9cac0aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): picked mon.noname-a con 0x7f9cac0adfc0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cac0a5ac0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0a9490 0x7f9cac0a9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac0a9490 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac0ac130 con 0x7f9cac0a5ac0 2023-08-22T23:22:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac0abf30 con 0x7f9cac0adfc0 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0ae3a0 con 0x7f9cac0a9490 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7f9cb2857640 10 monclient(hunting): authenticate will time out at 1516.788112s 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7fae42c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7fae42c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.537+0000 7fae42c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7ffb6affd640 10 monclient(hunting): get_auth_request con 0x7ffb6c0aa1a0 auth_method 0 2023-08-22T23:22:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7ffb6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7ffb6affd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7ffb6affd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.538+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-08-22T23:22:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 1 -- 172.21.15.29:0/935910049 >> v2:172.21.15.52:3300/0 conn(0x7fae3c0a7f80 msgr2=0x7fae3c0a8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 1 --2- 172.21.15.29:0/935910049 >> v2:172.21.15.52:3300/0 conn(0x7fae3c0a7f80 0x7fae3c0a8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 1 -- 172.21.15.29:0/935910049 >> v1:172.21.15.29:6789/0 conn(0x7fae3c0a7a10 legacy=0x7fae3c0afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 1 monclient: found mon.noname-b 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.540+0000 7fae42c76640 1 -- 172.21.15.29:0/935910049 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c141430 con 0x7fae3c0b0920 2023-08-22T23:22:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.541+0000 7ffb6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.541+0000 7ffb6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.541+0000 7ffb6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.542+0000 7ff491931640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ff48c0aafd0 0x7ff48c0adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 1 -- 172.21.15.29:0/3997040306 >> v1:172.21.15.52:6789/0 conn(0x7f6ac40a9e70 legacy=0x7f6ac40ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 1 -- 172.21.15.29:0/3997040306 >> v1:172.21.15.29:6789/0 conn(0x7f6ac40abc70 legacy=0x7f6ac413d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 1 monclient: found mon.noname-b 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.543+0000 7f6ac3fff640 1 -- 172.21.15.29:0/3997040306 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac40aa5f0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332022344 0 0) 0x7f0bd80717e0 con 0x7f0bd81457c0 2023-08-22T23:22:32.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7f0bd559a640 10 monclient(hunting): _negotiate 2023-08-22T23:22:32.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7f0bd559a640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7f0bd559a640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7f0bd559a640 10 monclient(hunting): my global_id is 4287 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc0003520 con 0x7f0bd81457c0 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7ff491931640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ff48c0aafd0 0x7ff48c0adf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39546/0 (socket says 172.21.15.29:39546) 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7ff491931640 1 -- 172.21.15.29:0/702184781 learned_addr learned my addr 172.21.15.29:0/702184781 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7ff491931640 10 monclient(hunting): get_auth_request con 0x7ff48c0aafd0 auth_method 0 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7ff491931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7ff491931640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.544+0000 7ff491931640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648416919 0 0) 0x7ffb6c0a91f0 con 0x7ffb6c0aafd0 2023-08-22T23:22:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): _negotiate 2023-08-22T23:22:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): my global_id is 4290 2023-08-22T23:22:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54003520 con 0x7ffb6c0aafd0 2023-08-22T23:22:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ff491931640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ff491931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ff491931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117123154 0 0) 0x7ffb6c0abe30 con 0x7ffb6c0adf30 2023-08-22T23:22:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): _negotiate 2023-08-22T23:22:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 10 monclient(hunting): my global_id is 4282 2023-08-22T23:22:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.545+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c0a91f0 con 0x7ffb6c0adf30 2023-08-22T23:22:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.546+0000 7f0bd5d9b640 1 --2- 172.21.15.29:0/962841963 >> v2:172.21.15.52:3300/0 conn(0x7f0bd8142d60 0x7f0bd80712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.546+0000 7f0bd5d9b640 10 monclient(hunting): get_auth_request con 0x7f0bd8142d60 auth_method 0 2023-08-22T23:22:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.546+0000 7f0bd5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.546+0000 7f0bd5d9b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.546+0000 7f0bd5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.546+0000 7f6ac3fff640 1 --2- 172.21.15.29:0/3997040306 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 0x7f6ac40a86d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009e80 tx=0x7f6abc02be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02256b7dc50af95 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.547+0000 7f0bd5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.547+0000 7f0bd5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.547+0000 7f0bd5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 1 -- 172.21.15.29:0/4100665875 >> v1:172.21.15.52:6789/0 conn(0x7ffb6c0adf30 legacy=0x7ffb6c1500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 1 -- 172.21.15.29:0/4100665875 >> v1:172.21.15.29:6789/0 conn(0x7ffb6c0aafd0 legacy=0x7ffb6c1491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 1 monclient: found mon.noname-b 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.548+0000 7ffb6affd640 1 -- 172.21.15.29:0/4100665875 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c0ac110 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7fae42c76640 1 --2- 172.21.15.29:0/935910049 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0b0920 0x7fae3c140ef0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fae380097a0 tx=0x7fae3802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6352d1fc176654f8 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2023-08-22T23:22:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 1 -- 172.21.15.29:0/702184781 >> v1:172.21.15.52:6789/0 conn(0x7ff48c149380 legacy=0x7ff48c1500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 1 -- 172.21.15.29:0/702184781 >> v1:172.21.15.29:6789/0 conn(0x7ff48c0aa1a0 legacy=0x7ff48c148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 1 monclient: found mon.noname-c 2023-08-22T23:22:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7ff491931640 1 -- 172.21.15.29:0/702184781 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c0ac110 con 0x7ff48c0aafd0 2023-08-22T23:22:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f7801334640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f77f4034aa0 0x7f77f402a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170073969 0 0) 0x7f0bc0003520 con 0x7f0bd81457c0 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 >> v2:172.21.15.52:3300/0 conn(0x7f0bd8142d60 msgr2=0x7f0bd80712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 --2- 172.21.15.29:0/962841963 >> v2:172.21.15.52:3300/0 conn(0x7f0bd8142d60 0x7f0bd80712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 >> v1:172.21.15.52:6789/0 conn(0x7f0bd80ae150 legacy=0x7f0bd8142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 monclient: found mon.noname-a-legacy 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.29:6789/0 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8071a30 con 0x7f0bd81457c0 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4066382038 0 0) 0x7f0bd0001670 con 0x7f0bd81457c0 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-08-22T23:22:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: dump: 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] but i'm connected to v1:172.21.15.29:6789/0, reconnecting 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 >> v1:172.21.15.29:6789/0 conn(0x7f0bd81457c0 legacy=0x7f0bd8145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bc0003970 0x7f0bc0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): picked mon.c con 0x7f0bc0003970 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bc00047c0 0x7f0bc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): picked mon.a con 0x7f0bc00047c0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc0005180 0x7f0bc000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): picked mon.b con 0x7f0bc0005180 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd81431a0 con 0x7f0bc00047c0 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc000c020 con 0x7f0bc0005180 2023-08-22T23:22:32.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc000c1e0 con 0x7f0bc0003970 2023-08-22T23:22:32.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.549+0000 7f0bd559a640 10 monclient(hunting): _finish_auth 0 2023-08-22T23:22:32.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.550+0000 7f7801334640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f77f4034aa0 0x7f77f402a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39552/0 (socket says 172.21.15.29:39552) 2023-08-22T23:22:32.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.550+0000 7f7801334640 1 -- 172.21.15.29:0/1654720128 learned_addr learned my addr 172.21.15.29:0/1654720128 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.550+0000 7f7801334640 10 monclient(hunting): get_auth_request con 0x7f77f4034aa0 auth_method 0 2023-08-22T23:22:32.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.550+0000 7f7801334640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.550+0000 7f7801334640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.550+0000 7f7801334640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.552+0000 7f42f77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f80aa510 0x7f42f80a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.553+0000 7f42f77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f80aa510 0x7f42f80a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33332/0 (socket says 172.21.15.29:33332) 2023-08-22T23:22:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.553+0000 7f42f77fe640 1 -- 172.21.15.29:0/1882235064 learned_addr learned my addr 172.21.15.29:0/1882235064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.553+0000 7f42f77fe640 10 monclient(hunting): get_auth_request con 0x7f42f80aa510 auth_method 0 2023-08-22T23:22:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.553+0000 7f42f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.553+0000 7f42f77fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.553+0000 7f42f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.554+0000 7f9cabfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cac0adfc0 0x7f9cac0aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.554+0000 7f42f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.554+0000 7f42f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.554+0000 7f42f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.555+0000 7ffb6affd640 1 --2- 172.21.15.29:0/4100665875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0aa1a0 0x7ffb6c148c70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009dc0 tx=0x7ffb6402bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ef6b994344a0a6f server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.556+0000 7f9cabfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cac0adfc0 0x7f9cac0aaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33342/0 (socket says 172.21.15.29:33342) 2023-08-22T23:22:32.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.556+0000 7f9cabfff640 1 -- 172.21.15.29:0/3835849822 learned_addr learned my addr 172.21.15.29:0/3835849822 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.556+0000 7f9cabfff640 10 monclient(hunting): get_auth_request con 0x7f9cac0adfc0 auth_method 0 2023-08-22T23:22:32.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.556+0000 7f9cabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.556+0000 7f9cabfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.556+0000 7f9cabfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2023-08-22T23:22:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 1 -- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f8071750 msgr2=0x7f42f80ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 1 --2- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f8071750 0x7f42f80ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 1 -- 172.21.15.29:0/1882235064 >> v1:172.21.15.29:6789/0 conn(0x7f42f80a8900 legacy=0x7f42f813e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 1 monclient: found mon.noname-a 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.557+0000 7f42f77fe640 1 -- 172.21.15.29:0/1882235064 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f80abc10 con 0x7f42f80aa510 2023-08-22T23:22:32.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f9cabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f9cabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f9cabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 1 -- 172.21.15.29:0/3997040306 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc02c9c0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: dump: 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 1 -- 172.21.15.29:0/3997040306 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc02cb20 con 0x7f6ac40b2df0 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 1 -- 172.21.15.29:0/3997040306 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc0343f0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac997f640 5 monclient: authenticate success, global_id 4250 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac997f640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac997f640 4 set_mon_vals no callback set 2023-08-22T23:22:32.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.558+0000 7f6ac37fe640 10 monclient: dump: 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.612 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 10 monclient: shutdown 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 -- 172.21.15.29:0/3997040306 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 msgr2=0x7f6ac40a86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 --2- 172.21.15.29:0/3997040306 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 0x7f6ac40a86d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009e80 tx=0x7f6abc02be20 comp rx=0 tx=0).stop 2023-08-22T23:22:32.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f0bd659c640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bc0003970 0x7f0bc0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 -- 172.21.15.29:0/3997040306 shutdown_connections 2023-08-22T23:22:32.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 --2- 172.21.15.29:0/3997040306 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40b2df0 0x7f6ac40a86d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 -- 172.21.15.29:0/3997040306 >> 172.21.15.29:0/3997040306 conn(0x7f6ac40b29e0 msgr2=0x7f6ac40aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 -- 172.21.15.29:0/3997040306 shutdown_connections 2023-08-22T23:22:32.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.559+0000 7f6ac997f640 1 -- 172.21.15.29:0/3997040306 wait complete. 2023-08-22T23:22:32.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f0bd6d9d640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc0005180 0x7f0bc000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 1 -- 172.21.15.29:0/935910049 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3802f050 con 0x7fae3c0b0920 2023-08-22T23:22:32.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:32.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: dump: 2023-08-22T23:22:32.616 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.617 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 1 -- 172.21.15.29:0/935910049 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38031700 con 0x7fae3c0b0920 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae43477640 5 monclient: authenticate success, global_id 4247 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae43477640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:32.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae43477640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 1 -- 172.21.15.29:0/935910049 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38031a90 con 0x7fae3c0b0920 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae43477640 4 set_mon_vals no callback set 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae41473640 10 monclient: dump: 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.620 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 10 monclient: monmap: 2023-08-22T23:22:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f9cb0df4640 1 --2- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0a9490 0x7f9cac0a9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 1 librados: starting msgr at 2023-08-22T23:22:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 1 librados: starting objecter 2023-08-22T23:22:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 1 Processor -- start 2023-08-22T23:22:32.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 1 -- start start 2023-08-22T23:22:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 1 librados: setting wanted keys 2023-08-22T23:22:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 1 librados: calling monclient init 2023-08-22T23:22:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7f6ac997f640 10 monclient: init 2023-08-22T23:22:32.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae43477640 10 monclient: shutdown 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.560+0000 7fae43477640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 -- 172.21.15.29:0/935910049 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0b0920 msgr2=0x7fae3c140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 --2- 172.21.15.29:0/935910049 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0b0920 0x7fae3c140ef0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fae380097a0 tx=0x7fae3802b7a0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 -- 172.21.15.29:0/935910049 shutdown_connections 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 --2- 172.21.15.29:0/935910049 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0b0920 0x7fae3c140ef0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 --2- 172.21.15.29:0/935910049 >> v2:172.21.15.52:3300/0 conn(0x7fae3c0a7f80 0x7fae3c0a8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 -- 172.21.15.29:0/935910049 >> 172.21.15.29:0/935910049 conn(0x7fae3c0b67d0 msgr2=0x7fae3c0b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 -- 172.21.15.29:0/935910049 shutdown_connections 2023-08-22T23:22:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7fae43477640 1 -- 172.21.15.29:0/935910049 wait complete. 2023-08-22T23:22:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac40a9e70 0x7f6ac4126420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac40a9e70 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40abc70 0x7f6ac4126960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac40abc70 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ac40b2df0 0x7f6ac41d9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac40b2df0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac406d490 con 0x7f6ac40a9e70 2023-08-22T23:22:32.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac406e230 con 0x7f6ac40b2df0 2023-08-22T23:22:32.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4129380 con 0x7f6ac40abc70 2023-08-22T23:22:32.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f6ac997f640 10 monclient(hunting): authenticate will time out at 1516.812194s 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 1 -- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0a9490 msgr2=0x7f9cac0a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 1 --2- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0a9490 0x7f9cac0a9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 1 -- 172.21.15.29:0/3835849822 >> v1:172.21.15.29:6789/0 conn(0x7f9cac0a5ac0 legacy=0x7f9cac0ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 1 monclient: found mon.noname-a 2023-08-22T23:22:32.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.561+0000 7f9cabfff640 1 -- 172.21.15.29:0/3835849822 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac0ac410 con 0x7f9cac0adfc0 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 10 monclient: monmap: 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.631 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 1 librados: starting msgr at 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 1 librados: starting objecter 2023-08-22T23:22:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 1 Processor -- start 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 1 -- start start 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 1 librados: setting wanted keys 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 1 librados: calling monclient init 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.562+0000 7fae43477640 10 monclient: init 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae3c0a7a10 0x7fae3c1da4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): picked mon.noname-a con 0x7fae3c0a7a10 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0a7f80 0x7fae3c142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): picked mon.noname-c con 0x7fae3c0a7f80 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fae3c0b0920 0x7fae3c142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): picked mon.noname-b con 0x7fae3c0b0920 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1450f0 con 0x7fae3c0a7a10 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c139bf0 con 0x7fae3c0b0920 2023-08-22T23:22:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1da9f0 con 0x7fae3c0a7f80 2023-08-22T23:22:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.563+0000 7fae43477640 10 monclient(hunting): authenticate will time out at 1516.814171s 2023-08-22T23:22:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd6d9d640 10 monclient(hunting): get_auth_request con 0x7f0bc0005180 auth_method 0 2023-08-22T23:22:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd6d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:22:32.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f9cabfff640 1 --2- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0adfc0 0x7f9cac0aaaf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098a0 tx=0x7f9c9c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a00bf11e6279a4 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd659c640 10 monclient(hunting): get_auth_request con 0x7f0bc0003970 auth_method 0 2023-08-22T23:22:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd659c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.564+0000 7f0bd659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:22:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.565+0000 7f42f77fe640 1 --2- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f80aa510 0x7f42f80a9490 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f42e80098a0 tx=0x7f42e802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c8eaafca2c4a28 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f0bd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f0bd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f0bd659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-08-22T23:22:32.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f7801b35640 1 --2- 172.21.15.29:0/1654720128 >> v2:172.21.15.29:3300/0 conn(0x7f77f40c3d00 0x7f77f40c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f7801334640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f7801334640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f7801334640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.566+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-08-22T23:22:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 10 monclient(hunting): handle_auth_done global_id 4287 payload 293 2023-08-22T23:22:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 1 -- 172.21.15.29:0/962841963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc0005180 msgr2=0x7f0bc000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc0005180 0x7f0bc000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 1 -- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bc00047c0 msgr2=0x7f0bc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bc00047c0 0x7f0bc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 1 monclient: found mon.c 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd6d9d640 1 -- 172.21.15.29:0/962841963 reap_dead start 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 1 -- 172.21.15.29:0/962841963 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8071a30 con 0x7f0bc0003970 2023-08-22T23:22:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bd659c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bdc8aa640 5 monclient: authenticate success, global_id 4287 2023-08-22T23:22:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.568+0000 7f0bdc8aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6402ccc0 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: dump: 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6402ce20 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 1 -- 172.21.15.29:0/4100665875 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64034660 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb69ffb640 10 monclient: dump: 2023-08-22T23:22:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.647 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 5 monclient: authenticate success, global_id 4253 2023-08-22T23:22:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 4 set_mon_vals no callback set 2023-08-22T23:22:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): get_auth_request con 0x7f77f40c3d00 auth_method 0 2023-08-22T23:22:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7fae41c74640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0a7f80 0x7fae3c142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f7801b35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 10 monclient: shutdown 2023-08-22T23:22:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 1 -- 172.21.15.29:0/4100665875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0aa1a0 msgr2=0x7ffb6c148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7ffb7135d640 1 --2- 172.21.15.29:0/4100665875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0aa1a0 0x7ffb6c148c70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009dc0 tx=0x7ffb6402bf10 comp rx=0 tx=0).stop 2023-08-22T23:22:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.572+0000 7f6ac917e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ac40b2df0 0x7f6ac41d9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7ffb7135d640 1 -- 172.21.15.29:0/4100665875 shutdown_connections 2023-08-22T23:22:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7ffb7135d640 1 --2- 172.21.15.29:0/4100665875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0aa1a0 0x7ffb6c148c70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7ffb7135d640 1 -- 172.21.15.29:0/4100665875 >> 172.21.15.29:0/4100665875 conn(0x7ffb6c0b27e0 msgr2=0x7ffb6c0b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7ff491931640 1 --2- 172.21.15.29:0/702184781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aafd0 0x7ff48c0adf30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0098a0 tx=0x7ff47c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37fd13b5303e4fb2 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7ffb7135d640 1 -- 172.21.15.29:0/4100665875 shutdown_connections 2023-08-22T23:22:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7f0bd659c640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bc0003970 0x7f0bc0003d90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc001810 tx=0x7f0bc00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4a78fc7f706b4e7 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.573+0000 7ffb7135d640 1 -- 172.21.15.29:0/4100665875 wait complete. 2023-08-22T23:22:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 10 monclient: monmap: 2023-08-22T23:22:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.653 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac917e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ac40b2df0 0x7f6ac41d9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39568/0 (socket says 172.21.15.29:39568) 2023-08-22T23:22:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac917e640 1 -- 172.21.15.29:0/4192804943 learned_addr learned my addr 172.21.15.29:0/4192804943 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac917e640 10 monclient(hunting): get_auth_request con 0x7f6ac40b2df0 auth_method 0 2023-08-22T23:22:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac917e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac917e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac917e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f77f3fff640 1 -- 172.21.15.29:0/1654720128 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777828093 0 0) 0x7f77f4032f60 con 0x7f77f4034e80 2023-08-22T23:22:32.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f77f3fff640 10 monclient(hunting): _negotiate 2023-08-22T23:22:32.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f77f3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f77f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f77f3fff640 10 monclient(hunting): my global_id is 4294 2023-08-22T23:22:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f77f3fff640 1 -- 172.21.15.29:0/1654720128 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e8003440 con 0x7f77f4034e80 2023-08-22T23:22:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 1 librados: starting msgr at 2023-08-22T23:22:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 1 librados: starting objecter 2023-08-22T23:22:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7f6ac3fff640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40abc70 0x7f6ac4126960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7fae41c74640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0a7f80 0x7fae3c142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43380/0 (socket says 172.21.15.29:43380) 2023-08-22T23:22:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7fae41c74640 1 -- 172.21.15.29:0/1832659047 learned_addr learned my addr 172.21.15.29:0/1832659047 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7fae41c74640 10 monclient(hunting): get_auth_request con 0x7fae3c0a7f80 auth_method 0 2023-08-22T23:22:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7fae41c74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7fae41c74640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7fae41c74640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 1 Processor -- start 2023-08-22T23:22:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 1 -- start start 2023-08-22T23:22:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 1 librados: setting wanted keys 2023-08-22T23:22:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 1 librados: calling monclient init 2023-08-22T23:22:32.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.574+0000 7ffb7135d640 10 monclient: init 2023-08-22T23:22:32.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-08-22T23:22:32.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 1 -- 172.21.15.29:0/1654720128 >> v2:172.21.15.52:3300/0 conn(0x7f77f4034aa0 msgr2=0x7f77f402a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:32.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 1 --2- 172.21.15.29:0/1654720128 >> v2:172.21.15.52:3300/0 conn(0x7f77f4034aa0 0x7f77f402a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 1 -- 172.21.15.29:0/1654720128 >> v1:172.21.15.52:6789/0 conn(0x7f77f4034e80 legacy=0x7f77f4032850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 1 monclient: found mon.noname-a 2023-08-22T23:22:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f7801b35640 1 -- 172.21.15.29:0/1654720128 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f402cd10 con 0x7f77f40c3d00 2023-08-22T23:22:32.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:32.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb6c0aa1a0 0x7ffb6c1c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6c0aa1a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c0aafd0 0x7ffb6c1c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb6c0aafd0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0adf30 0x7ffb6c1da3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6c0adf30 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1c4df0 con 0x7ffb6c0aafd0 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1c6150 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1c63c0 con 0x7ffb6c0adf30 2023-08-22T23:22:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7ffb7135d640 10 monclient(hunting): authenticate will time out at 1516.825973s 2023-08-22T23:22:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f6ac897d640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac40a9e70 0x7f6ac4126420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7fae41c74640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7fae41c74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7fae41c74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f6ac3fff640 10 monclient(hunting): get_auth_request con 0x7f6ac40abc70 auth_method 0 2023-08-22T23:22:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f6ac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f6ac3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.575+0000 7f6ac3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.576+0000 7f6ac917e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.576+0000 7f6ac917e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.576+0000 7f6ac917e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.576+0000 7fae42c76640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fae3c0b0920 0x7fae3c142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7f6ac897d640 10 monclient(hunting): get_auth_request con 0x7f6ac40a9e70 auth_method 0 2023-08-22T23:22:32.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7f6ac897d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7f6ac897d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7f6ac897d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7fae42c76640 10 monclient(hunting): get_auth_request con 0x7fae3c0b0920 auth_method 0 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7fae42c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7fae42c76640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.577+0000 7fae42c76640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 1 -- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fae3c0b0920 msgr2=0x7fae3c142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fae3c0b0920 0x7fae3c142740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 1 -- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae3c0a7a10 msgr2=0x7fae3c1da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae3c0a7a10 0x7fae3c1da4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.579+0000 7fae41c74640 1 monclient: found mon.noname-c 2023-08-22T23:22:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.580+0000 7fae41c74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.580+0000 7fae41c74640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.580+0000 7fae41c74640 1 -- 172.21.15.29:0/1832659047 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009400 con 0x7fae3c0a7f80 2023-08-22T23:22:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.580+0000 7f6ac897d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.580+0000 7f6ac897d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.580+0000 7f6ac897d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.581+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.581+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.581+0000 7f6ac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 1 -- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40abc70 msgr2=0x7f6ac4126960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ac40abc70 0x7f6ac4126960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 1 -- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac40a9e70 msgr2=0x7f6ac4126420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac40a9e70 0x7f6ac4126420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 1 monclient: found mon.noname-b 2023-08-22T23:22:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.583+0000 7f6ac917e640 1 -- 172.21.15.29:0/4192804943 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc009b30 con 0x7f6ac40b2df0 2023-08-22T23:22:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.584+0000 7f7801b35640 1 --2- 172.21.15.29:0/1654720128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f40c3d00 0x7f77f40c4110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77fc072340 tx=0x7f77fc0a4e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89afb25cb5113d03 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 1 -- 172.21.15.29:0/702184781 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c031640 con 0x7ff48c0aafd0 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: dump: 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ffb6affd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb6c0aa1a0 0x7ffb6c1c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 1 -- 172.21.15.29:0/702184781 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c0317a0 con 0x7ff48c0aafd0 2023-08-22T23:22:32.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff494396640 5 monclient: authenticate success, global_id 4288 2023-08-22T23:22:32.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff494396640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:32.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff494396640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 1 -- 172.21.15.29:0/702184781 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c031b30 con 0x7ff48c0aafd0 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff494396640 4 set_mon_vals no callback set 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.586+0000 7ff491130640 10 monclient: dump: 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 10 monclient: shutdown 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 -- 172.21.15.29:0/702184781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aafd0 msgr2=0x7ff48c0adf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 --2- 172.21.15.29:0/702184781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aafd0 0x7ff48c0adf30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0098a0 tx=0x7ff47c02b8f0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 -- 172.21.15.29:0/702184781 shutdown_connections 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 --2- 172.21.15.29:0/702184781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aafd0 0x7ff48c0adf30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 -- 172.21.15.29:0/702184781 >> 172.21.15.29:0/702184781 conn(0x7ff48c0b27e0 msgr2=0x7ff48c0b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 -- 172.21.15.29:0/702184781 shutdown_connections 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.587+0000 7ff494396640 1 -- 172.21.15.29:0/702184781 wait complete. 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 10 monclient: monmap: 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 1 librados: starting msgr at 2023-08-22T23:22:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 1 librados: starting objecter 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 1 Processor -- start 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 1 -- start start 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 1 librados: setting wanted keys 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 1 librados: calling monclient init 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7ff494396640 10 monclient: init 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7f6ac917e640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ac40b2df0 0x7f6ac41d9470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c390 tx=0x7f6ab800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7f6ac17fa640 10 client.?.objecter ms_handle_connect 0x7f6ac40b2df0 2023-08-22T23:22:32.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.588+0000 7f6ac17fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:32.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aa1a0 0x7ff48c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c0aa1a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0aafd0 0x7ff48c06e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): picked mon.noname-a con 0x7ff48c0aafd0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff48c149380 0x7ff48c1daa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): picked mon.noname-c con 0x7ff48c149380 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c1da510 con 0x7ff48c0aafd0 2023-08-22T23:22:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c0a4d00 con 0x7ff48c0aa1a0 2023-08-22T23:22:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c0a4ec0 con 0x7ff48c149380 2023-08-22T23:22:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ff494396640 10 monclient(hunting): authenticate will time out at 1516.839658s 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ffb6affd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb6c0aa1a0 0x7ffb6c1c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39588/0 (socket says 172.21.15.29:39588) 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ffb6affd640 1 -- 172.21.15.29:0/868261135 learned_addr learned my addr 172.21.15.29:0/868261135 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ffb6affd640 10 monclient(hunting): get_auth_request con 0x7ffb6c0aa1a0 auth_method 0 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ffb6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ffb6affd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.589+0000 7ffb6affd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.590+0000 7ffb6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.590+0000 7ffb6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.590+0000 7ffb6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.590+0000 7fae41c74640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0a7f80 0x7fae3c142200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c400 tx=0x7fae2c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.590+0000 7fae32ffd640 10 client.?.objecter ms_handle_connect 0x7fae3c0a7f80 2023-08-22T23:22:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.590+0000 7fae32ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 1 -- 172.21.15.29:0/3835849822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0317c0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: dump: 2023-08-22T23:22:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9cb2857640 5 monclient: authenticate success, global_id 4296 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9cb2857640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 1 -- 172.21.15.29:0/3835849822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c031920 con 0x7f9cac0adfc0 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9cb2857640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9cb2857640 4 set_mon_vals no callback set 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 1 -- 172.21.15.29:0/3835849822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c031cb0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9caaffd640 10 monclient: dump: 2023-08-22T23:22:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.692 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 1 -- 172.21.15.29:0/1882235064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e80317c0 con 0x7f42f80aa510 2023-08-22T23:22:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: dump: 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 1 -- 172.21.15.29:0/1882235064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e8031920 con 0x7f42f80aa510 2023-08-22T23:22:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 1 -- 172.21.15.29:0/1882235064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8031cb0 con 0x7f42f80aa510 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42f6ffd640 10 monclient: dump: 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42fe6c5640 5 monclient: authenticate success, global_id 4293 2023-08-22T23:22:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42fe6c5640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f42fe6c5640 4 set_mon_vals no callback set 2023-08-22T23:22:32.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.591+0000 7f9cb2857640 10 monclient: shutdown 2023-08-22T23:22:32.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 -- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0adfc0 msgr2=0x7f9cac0aaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 --2- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0adfc0 0x7f9cac0aaaf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098a0 tx=0x7f9c9c02b860 comp rx=0 tx=0).stop 2023-08-22T23:22:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 -- 172.21.15.29:0/3835849822 shutdown_connections 2023-08-22T23:22:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 --2- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0a9490 0x7f9cac0a9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 --2- 172.21.15.29:0/3835849822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0adfc0 0x7f9cac0aaaf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 -- 172.21.15.29:0/3835849822 >> 172.21.15.29:0/3835849822 conn(0x7f9cac0aee40 msgr2=0x7f9cac0af220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 -- 172.21.15.29:0/3835849822 shutdown_connections 2023-08-22T23:22:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 10 monclient: shutdown 2023-08-22T23:22:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 -- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f80aa510 msgr2=0x7f42f80a9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 --2- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f80aa510 0x7f42f80a9490 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f42e80098a0 tx=0x7f42e802b860 comp rx=0 tx=0).stop 2023-08-22T23:22:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 -- 172.21.15.29:0/1882235064 shutdown_connections 2023-08-22T23:22:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 --2- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f80aa510 0x7f42f80a9490 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 --2- 172.21.15.29:0/1882235064 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f8071750 0x7f42f80ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 -- 172.21.15.29:0/1882235064 >> 172.21.15.29:0/1882235064 conn(0x7f42f80aee40 msgr2=0x7f42f80af220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc004aa0 con 0x7f0bc0003970 2023-08-22T23:22:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: dump: 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.702 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc02e070 con 0x7f0bc0003970 2023-08-22T23:22:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bdc8aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 -- 172.21.15.29:0/1882235064 shutdown_connections 2023-08-22T23:22:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f42fe6c5640 1 -- 172.21.15.29:0/1882235064 wait complete. 2023-08-22T23:22:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 1 -- 172.21.15.29:0/962841963 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc004e00 con 0x7f0bc0003970 2023-08-22T23:22:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bd559a640 10 monclient: dump: 2023-08-22T23:22:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.705 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bdc8aa640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f0bdc8aa640 4 set_mon_vals no callback set 2023-08-22T23:22:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.592+0000 7f9cb2857640 1 -- 172.21.15.29:0/3835849822 wait complete. 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 10 monclient: monmap: 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 1 librados: starting msgr at 2023-08-22T23:22:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 1 librados: starting objecter 2023-08-22T23:22:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 10 monclient: shutdown 2023-08-22T23:22:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bc0003970 msgr2=0x7f0bc0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bc0003970 0x7f0bc0003d90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc001810 tx=0x7f0bc00057f0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 10 monclient: monmap: 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 1 librados: starting msgr at 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 1 librados: starting objecter 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 1 Processor -- start 2023-08-22T23:22:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 1 Processor -- start 2023-08-22T23:22:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/962841963 shutdown_connections 2023-08-22T23:22:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 --2- 172.21.15.29:0/962841963 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bc0003970 0x7f0bc0003d90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/962841963 >> 172.21.15.29:0/962841963 conn(0x7f0bd80afad0 msgr2=0x7f0bd80b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/962841963 shutdown_connections 2023-08-22T23:22:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 1 -- start start 2023-08-22T23:22:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 1 librados: setting wanted keys 2023-08-22T23:22:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 1 librados: calling monclient init 2023-08-22T23:22:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f9cb2857640 10 monclient: init 2023-08-22T23:22:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/962841963 wait complete. 2023-08-22T23:22:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 1 -- start start 2023-08-22T23:22:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 1 librados: setting wanted keys 2023-08-22T23:22:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 1 librados: calling monclient init 2023-08-22T23:22:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.593+0000 7f42fe6c5640 10 monclient: init 2023-08-22T23:22:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8071750 0x7f42f8133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): picked mon.noname-a con 0x7f42f8071750 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f80a8900 0x7f42f80a77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): picked mon.noname-c con 0x7f42f80a8900 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42f8138710 0x7f42f813a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): picked mon.noname-b con 0x7f42f8138710 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f813a2b0 con 0x7f42f8071750 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80a80f0 con 0x7f42f8138710 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81381f0 con 0x7f42f80a8900 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f42fe6c5640 10 monclient(hunting): authenticate will time out at 1516.845087s 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9cac0a5ac0 0x7f9cac070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac0a5ac0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0a9490 0x7f9cac0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): picked mon.noname-a con 0x7f9cac0a9490 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0adfc0 0x7f9cac1da4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): picked mon.noname-c con 0x7f9cac0adfc0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0703c0 con 0x7f9cac0a9490 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac1d9fd0 con 0x7f9cac0a5ac0 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac06e4e0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f9cb2857640 10 monclient(hunting): authenticate will time out at 1516.845098s 2023-08-22T23:22:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f0bdc8aa640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.594+0000 7f0bdc8aa640 10 monclient: monmap: 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 1 librados: starting msgr at 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 1 librados: starting objecter 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 1 Processor -- start 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 1 -- start start 2023-08-22T23:22:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 1 librados: setting wanted keys 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 1 librados: calling monclient init 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.595+0000 7f0bdc8aa640 10 monclient: init 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bc0003970 0x7f0bd81485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc0003970 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bd80a8ee0 0x7f0bd80a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd80a8ee0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bd81da340 0x7f0bd81da7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd81da340 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd8149330 con 0x7f0bc0003970 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd80a9960 con 0x7f0bd80a8ee0 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd81dad00 con 0x7f0bd81da340 2023-08-22T23:22:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7f0bdc8aa640 10 monclient(hunting): authenticate will time out at 1516.846582s 2023-08-22T23:22:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2023-08-22T23:22:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 1 -- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0adf30 msgr2=0x7ffb6c1da3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffb6c0adf30 0x7ffb6c1da3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 1 -- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c0aafd0 msgr2=0x7ffb6c1c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c0aafd0 0x7ffb6c1c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 1 monclient: found mon.noname-b 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.596+0000 7ffb6affd640 1 -- 172.21.15.29:0/868261135 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64009a70 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0aafd0 0x7ff48c06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0aafd0 0x7ff48c06e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33406/0 (socket says 172.21.15.29:33406) 2023-08-22T23:22:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 1 -- 172.21.15.29:0/235105092 learned_addr learned my addr 172.21.15.29:0/235105092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 10 monclient(hunting): get_auth_request con 0x7ff48c0aafd0 auth_method 0 2023-08-22T23:22:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ff491931640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ffb6affd640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb6c0aa1a0 0x7ffb6c1c8e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb64000f80 tx=0x7ffb64004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ffb537fe640 10 client.?.objecter ms_handle_connect 0x7ffb6c0aa1a0 2023-08-22T23:22:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.598+0000 7ffb537fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.599+0000 7ff491931640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.599+0000 7ff491931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.599+0000 7ff491931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.599+0000 7ff492132640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aa1a0 0x7ff48c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.600+0000 7ff492132640 10 monclient(hunting): get_auth_request con 0x7ff48c0aa1a0 auth_method 0 2023-08-22T23:22:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.600+0000 7ff492132640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.600+0000 7ff492132640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.600+0000 7ff492132640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.600+0000 7f9cb0df4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0adfc0 0x7f9cac1da4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.601+0000 7ff492132640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.601+0000 7ff492132640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.601+0000 7ff492132640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.601+0000 7f42fcc62640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42f8138710 0x7f42f813a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 1 -- 172.21.15.29:0/235105092 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff48c149380 msgr2=0x7ff48c1daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff48c149380 0x7ff48c1daa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 1 -- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aa1a0 msgr2=0x7ff48c06d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff48c0aa1a0 0x7ff48c06d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 1 monclient: found mon.noname-a 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7ff491931640 1 -- 172.21.15.29:0/235105092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c0094d0 con 0x7ff48c0aafd0 2023-08-22T23:22:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7f9cb0df4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0adfc0 0x7f9cac1da4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43408/0 (socket says 172.21.15.29:43408) 2023-08-22T23:22:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7f9cb0df4640 1 -- 172.21.15.29:0/2957632360 learned_addr learned my addr 172.21.15.29:0/2957632360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7f9cb0df4640 10 monclient(hunting): get_auth_request con 0x7f9cac0adfc0 auth_method 0 2023-08-22T23:22:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7f9cb0df4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7f9cb0df4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.602+0000 7f9cb0df4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.603+0000 7f42fcc62640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42f8138710 0x7f42f813a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39622/0 (socket says 172.21.15.29:39622) 2023-08-22T23:22:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.603+0000 7f42fcc62640 1 -- 172.21.15.29:0/464047534 learned_addr learned my addr 172.21.15.29:0/464047534 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.603+0000 7f42fcc62640 10 monclient(hunting): get_auth_request con 0x7f42f8138710 auth_method 0 2023-08-22T23:22:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.603+0000 7f42fcc62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.603+0000 7f42fcc62640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.603+0000 7f42fcc62640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80035f0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f9cabfff640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9cac0a5ac0 0x7f9cac070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: dump: 2023-08-22T23:22:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8003750 con 0x7f6ac40b2df0 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 5 monclient: authenticate success, global_id 4300 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 10 monclient: _renew_subs 2023-08-22T23:22:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac413e180 con 0x7f6ac40b2df0 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8003b00 con 0x7f6ac40b2df0 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac17fa640 10 monclient: dump: 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac2ffd640 4 set_mon_vals no callback set 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 10 monclient: _renew_subs 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac406d0b0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f6ac997f640 1 librados: init done 2023-08-22T23:22:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0052d0 con 0x7fae3c0a7f80 2023-08-22T23:22:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: dump: 2023-08-22T23:22:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c005430 con 0x7fae3c0a7f80 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c003830 con 0x7fae3c0a7f80 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae32ffd640 10 monclient: dump: 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae40c72640 4 set_mon_vals no callback set 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 5 monclient: authenticate success, global_id 4256 2023-08-22T23:22:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 10 monclient: _renew_subs 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c144c80 con 0x7fae3c0a7f80 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 10 monclient: _renew_subs 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c1d9e30 con 0x7fae3c0a7f80 2023-08-22T23:22:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7fae43477640 1 librados: init done 2023-08-22T23:22:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f42fcc62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f42fcc62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.604+0000 7f42fcc62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bd81da340 0x7f0bd81da7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bd81da340 0x7f0bd81da7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43422/0 (socket says 172.21.15.29:43422) 2023-08-22T23:22:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 1 -- 172.21.15.29:0/4122970911 learned_addr learned my addr 172.21.15.29:0/4122970911 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 10 monclient(hunting): get_auth_request con 0x7f0bd81da340 auth_method 0 2023-08-22T23:22:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f0bd6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f6ac997f640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f9cb0df4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f9cb0df4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f9cb0df4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f6ac997f640 15 client.4300.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f6ac997f640 20 client.4300.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f6ac997f640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7fae43477640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f6ac997f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7fae43477640 15 client.4256.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7fae43477640 20 client.4256.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7fae43477640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7fae43477640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae3c144f30 con 0x7fae3c0a7f80 2023-08-22T23:22:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.605+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ac40aa8d0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 1 -- 172.21.15.29:0/1654720128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0a5cd0 con 0x7f77f40c3d00 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: dump: 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 1 -- 172.21.15.29:0/1654720128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc0a5e70 con 0x7f77f40c3d00 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 1 -- 172.21.15.29:0/1654720128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0ad640 con 0x7f77f40c3d00 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f77f3fff640 10 monclient: dump: 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.756 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f7802336640 5 monclient: authenticate success, global_id 4299 2023-08-22T23:22:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f7802336640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.606+0000 7f7802336640 4 set_mon_vals no callback set 2023-08-22T23:22:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f9cab7fe640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0a9490 0x7f9cac0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 10 monclient: shutdown 2023-08-22T23:22:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 -- 172.21.15.29:0/1654720128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f40c3d00 msgr2=0x7f77f40c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 --2- 172.21.15.29:0/1654720128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f40c3d00 0x7f77f40c4110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77fc072340 tx=0x7f77fc0a4e60 comp rx=0 tx=0).stop 2023-08-22T23:22:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f9cab7fe640 10 monclient(hunting): get_auth_request con 0x7f9cac0a9490 auth_method 0 2023-08-22T23:22:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f9cab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f9cab7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f9cab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 -- 172.21.15.29:0/1654720128 shutdown_connections 2023-08-22T23:22:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 --2- 172.21.15.29:0/1654720128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f40c3d00 0x7f77f40c4110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 --2- 172.21.15.29:0/1654720128 >> v2:172.21.15.52:3300/0 conn(0x7f77f4034aa0 0x7f77f402a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 -- 172.21.15.29:0/1654720128 >> 172.21.15.29:0/1654720128 conn(0x7f77f40060a0 msgr2=0x7f77f40064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 -- 172.21.15.29:0/1654720128 shutdown_connections 2023-08-22T23:22:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.607+0000 7f7802336640 1 -- 172.21.15.29:0/1654720128 wait complete. 2023-08-22T23:22:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7f7802336640 10 monclient: build_initial_monmap 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7f7802336640 10 monclient: monmap: 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7f7802336640 1 librados: starting msgr at 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7f7802336640 1 librados: starting objecter 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7f7802336640 1 Processor -- start 2023-08-22T23:22:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cabfff640 10 monclient(hunting): get_auth_request con 0x7f9cac0a5ac0 auth_method 0 2023-08-22T23:22:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cabfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cabfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7ff491931640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0aafd0 0x7ff48c06e520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff47c000c00 tx=0x7ff47c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.608+0000 7ff482ffd640 10 client.?.objecter ms_handle_connect 0x7ff48c0aafd0 2023-08-22T23:22:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7ff482ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 -- start start 2023-08-22T23:22:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 librados: setting wanted keys 2023-08-22T23:22:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 librados: calling monclient init 2023-08-22T23:22:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient: init 2023-08-22T23:22:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f4034aa0 0x7f77f4031870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): picked mon.noname-a con 0x7f77f4034aa0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77f40c3d00 0x7f77f40c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): picked mon.noname-b con 0x7f77f40c3d00 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f77f40c6060 0x7f77f40c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): picked mon.noname-c con 0x7f77f40c6060 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f40c7fd0 con 0x7f77f4034aa0 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f40c7810 con 0x7f77f40c3d00 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f40c7ab0 con 0x7f77f40c6060 2023-08-22T23:22:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f7802336640 10 monclient(hunting): authenticate will time out at 1516.860159s 2023-08-22T23:22:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2023-08-22T23:22:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 1 -- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9cac0a5ac0 msgr2=0x7f9cac070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9cac0a5ac0 0x7f9cac070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 1 -- 172.21.15.29:0/2957632360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0a9490 msgr2=0x7f9cac0a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cac0a9490 0x7f9cac0a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 1 monclient: found mon.noname-c 2023-08-22T23:22:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.609+0000 7f9cb0df4640 1 -- 172.21.15.29:0/2957632360 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0094d0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6403e610 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: dump: 2023-08-22T23:22:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6403e770 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 5 monclient: authenticate success, global_id 4306 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 10 monclient: _renew_subs 2023-08-22T23:22:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c1c4a80 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6403eb20 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb537fe640 10 monclient: dump: 2023-08-22T23:22:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 10 monclient: _renew_subs 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c1cd6a0 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb7135d640 1 librados: init done 2023-08-22T23:22:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7ffb697fa640 4 set_mon_vals no callback set 2023-08-22T23:22:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.610+0000 7f42f7fff640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8071750 0x7f42f8133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.611+0000 7ffb7135d640 10 client.4306.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.611+0000 7ffb7135d640 15 client.4306.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.611+0000 7ffb7135d640 20 client.4306.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.611+0000 7ffb7135d640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.611+0000 7ffb7135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.611+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb6c0ac6d0 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2023-08-22T23:22:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 1 -- 172.21.15.29:0/464047534 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f80a8900 msgr2=0x7f42f80a77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f42f80a8900 0x7f42f80a77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 1 -- 172.21.15.29:0/464047534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8071750 msgr2=0x7f42f8133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8071750 0x7f42f8133d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 1 monclient: found mon.noname-b 2023-08-22T23:22:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.612+0000 7f42fcc62640 1 -- 172.21.15.29:0/464047534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e80094d0 con 0x7f42f8138710 2023-08-22T23:22:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.613+0000 7f0bd5d9b640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bd80a8ee0 0x7f0bd80a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2023-08-22T23:22:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 1 -- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bd80a8ee0 msgr2=0x7f0bd80a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bd80a8ee0 0x7f0bd80a9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 1 -- 172.21.15.29:0/4122970911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bc0003970 msgr2=0x7f0bd81485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bc0003970 0x7f0bd81485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 1 monclient: found mon.noname-c 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.614+0000 7f0bd6d9d640 1 -- 172.21.15.29:0/4122970911 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd81daec0 con 0x7f0bd81da340 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.617+0000 7f42fcc62640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42f8138710 0x7f42f813a630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c390 tx=0x7f42ec00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.617+0000 7f42f4ff9640 10 client.?.objecter ms_handle_connect 0x7f42f8138710 2023-08-22T23:22:32.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.617+0000 7f42f4ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.618+0000 7f9cb0df4640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0adfc0 0x7f9cac1da4f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400cd70 tx=0x7f9ca400a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.618+0000 7f9ca8ff9640 10 client.?.objecter ms_handle_connect 0x7f9cac0adfc0 2023-08-22T23:22:32.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.618+0000 7f9ca8ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.621+0000 7f7800b33640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77f40c3d00 0x7f77f40c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.621+0000 7f0bd6d9d640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bd81da340 0x7f0bd81da7c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80b34f0 tx=0x7f0bd000fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.621+0000 7f0bc6ffd640 10 client.?.objecter ms_handle_connect 0x7f0bd81da340 2023-08-22T23:22:32.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.621+0000 7f0bc6ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.622+0000 7f7801b35640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f77f40c6060 0x7f77f40c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c03ac60 con 0x7ff48c0aafd0 2023-08-22T23:22:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: dump: 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.784 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f4034aa0 0x7f77f4031870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c03ade0 con 0x7ff48c0aafd0 2023-08-22T23:22:32.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c044630 con 0x7ff48c0aafd0 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 5 monclient: authenticate success, global_id 4305 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 10 monclient: _renew_subs 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c06ea60 con 0x7ff48c0aafd0 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff49092f640 4 set_mon_vals no callback set 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff482ffd640 10 monclient: dump: 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 10 monclient: _renew_subs 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c070270 con 0x7ff48c0aafd0 2023-08-22T23:22:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7ff494396640 1 librados: init done 2023-08-22T23:22:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f4034aa0 0x7f77f4031870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33438/0 (socket says 172.21.15.29:33438) 2023-08-22T23:22:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 1 -- 172.21.15.29:0/1192927987 learned_addr learned my addr 172.21.15.29:0/1192927987 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 10 monclient(hunting): get_auth_request con 0x7f77f4034aa0 auth_method 0 2023-08-22T23:22:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801334640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801b35640 10 monclient(hunting): get_auth_request con 0x7f77f40c6060 auth_method 0 2023-08-22T23:22:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801b35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801b35640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.623+0000 7f7801b35640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.624+0000 7ff494396640 10 client.4305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.624+0000 7ff494396640 15 client.4305.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.624+0000 7ff494396640 20 client.4305.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.624+0000 7ff494396640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.624+0000 7ff494396640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.624+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff48c0ac6d0 con 0x7ff48c0aafd0 2023-08-22T23:22:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.625+0000 7f7801334640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.625+0000 7f7801334640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.625+0000 7f7801334640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.626+0000 7f7801b35640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.626+0000 7f7801b35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.626+0000 7f7801b35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0035f0 con 0x7f42f8138710 2023-08-22T23:22:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: dump: 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec003750 con 0x7f42f8138710 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 5 monclient: authenticate success, global_id 4312 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 10 monclient: _renew_subs 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 1 -- 172.21.15.29:0/464047534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f813bc40 con 0x7f42f8138710 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec003b00 con 0x7f42f8138710 2023-08-22T23:22:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f4ff9640 10 monclient: dump: 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 10 monclient: _renew_subs 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42f67fc640 4 set_mon_vals no callback set 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 1 -- 172.21.15.29:0/464047534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f813adb0 con 0x7f42f8138710 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.627+0000 7f42fe6c5640 1 librados: init done 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.628+0000 7f42fe6c5640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.628+0000 7f42fe6c5640 15 client.4312.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.628+0000 7f42fe6c5640 20 client.4312.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.628+0000 7f42fe6c5640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.628+0000 7f42fe6c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.628+0000 7f42fe6c5640 1 -- 172.21.15.29:0/464047534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f42f8138410 con 0x7f42f8138710 2023-08-22T23:22:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.631+0000 7f7801b35640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2023-08-22T23:22:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.631+0000 7f7801b35640 1 -- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77f40c3d00 msgr2=0x7f77f40c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.631+0000 7f7801b35640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77f40c3d00 0x7f77f40c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.631+0000 7f7801b35640 1 -- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f4034aa0 msgr2=0x7f77f4031870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.632+0000 7f7801b35640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77f4034aa0 0x7f77f4031870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.632+0000 7f7801b35640 10 monclient: _finish_hunting 0 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.632+0000 7f7801b35640 1 monclient: found mon.noname-c 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.632+0000 7f7801b35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.632+0000 7f7801b35640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.632+0000 7f7801b35640 1 -- 172.21.15.29:0/1192927987 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc066500 con 0x7f77f40c6060 2023-08-22T23:22:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4003b90 con 0x7f9cac0adfc0 2023-08-22T23:22:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: dump: 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca4003cf0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4004240 con 0x7f9cac0adfc0 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 5 monclient: authenticate success, global_id 4262 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 10 monclient: _renew_subs 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac0a2030 con 0x7f9cac0adfc0 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9caa7fc640 4 set_mon_vals no callback set 2023-08-22T23:22:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9ca8ff9640 10 monclient: dump: 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 10 monclient: _renew_subs 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac1daa30 con 0x7f9cac0adfc0 2023-08-22T23:22:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.642+0000 7f9cb2857640 1 librados: init done 2023-08-22T23:22:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f9cb2857640 10 client.4262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f9cb2857640 15 client.4262.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f9cb2857640 20 client.4262.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f9cb2857640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f9cb2857640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9cac0673f0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd000fe50 con 0x7f0bd81da340 2023-08-22T23:22:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: dump: 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd00052e0 con 0x7f0bd81da340 2023-08-22T23:22:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd000d8c0 con 0x7f0bd81da340 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 5 monclient: authenticate success, global_id 4265 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 10 monclient: _renew_subs 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/4122970911 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd81db050 con 0x7f0bd81da340 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bd4d99640 4 set_mon_vals no callback set 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bc6ffd640 10 monclient: dump: 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 10 monclient: _renew_subs 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/4122970911 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd80a76c0 con 0x7f0bd81da340 2023-08-22T23:22:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.643+0000 7f0bdc8aa640 1 librados: init done 2023-08-22T23:22:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.644+0000 7f0bdc8aa640 10 client.4265.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.644+0000 7f0bdc8aa640 15 client.4265.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.644+0000 7f0bdc8aa640 20 client.4265.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.644+0000 7f0bdc8aa640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.644+0000 7f0bdc8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.644+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/4122970911 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0bd80ac8d0 con 0x7f0bd81da340 2023-08-22T23:22:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.645+0000 7f7801b35640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f77f40c6060 0x7f77f40c55c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77fc04f3b0 tx=0x7f77fc072a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.645+0000 7f77f1ffb640 10 client.?.objecter ms_handle_connect 0x7f77f40c6060 2023-08-22T23:22:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.645+0000 7f77f1ffb640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.650+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff47c02f050 con 0x7ff48c0aafd0 2023-08-22T23:22:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.651+0000 7ff482ffd640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff47006ba30 0x7ff47006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.651+0000 7ff492132640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff47006ba30 0x7ff47006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.651+0000 7ff492132640 10 monclient: get_auth_request con 0x7ff47006ba30 auth_method 0 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.652+0000 7ff492132640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff47006ba30 0x7ff47006def0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff488004f60 tx=0x7ff488007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.652+0000 7ff482ffd640 10 client.4305.objecter ms_handle_connect 0x7ff47006ba30 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff47c07dbb0 con 0x7ff48c0aafd0 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 10 client.4305.objecter ms_dispatch 0x7ff48c000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 3 client.4305.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 3 client.4305.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 10 client.4305.objecter checking command 1 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7ff470071e10 0x7ff470074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 20 client.4305.objecter _get_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 15 client.4305.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 20 client.4305.objecter _get_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 20 client.4305.objecter get_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 15 client.4305.objecter _session_command_op_assign 7 1 2023-08-22T23:22:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 4 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 10 client.4305.objecter _send_command 1 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 --> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4700748e0 con 0x7ff470071e10 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 20 client.4305.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff47c0761e0 con 0x7ff48c0aafd0 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff482ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.656+0000 7ff492933640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7ff470071e10 0x7ff470074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.657+0000 7ff492933640 10 monclient: get_auth_request con 0x7ff470071e10 auth_method 0 2023-08-22T23:22:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.658+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ffb6403b050 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.658+0000 7ffb537fe640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffb5806b9e0 0x7ffb5806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.659+0000 7ffb6a7fc640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffb5806b9e0 0x7ffb5806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.659+0000 7ffb6a7fc640 10 monclient: get_auth_request con 0x7ffb5806b9e0 auth_method 0 2023-08-22T23:22:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.659+0000 7ffb537fe640 10 client.4306.objecter ms_handle_connect 0x7ffb5806b9e0 2023-08-22T23:22:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.659+0000 7ffb6a7fc640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffb5806b9e0 0x7ffb5806dea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffb600010e0 tx=0x7ffb600078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.660+0000 7ff482ffd640 10 client.4305.objecter ms_handle_connect 0x7ff470071e10 2023-08-22T23:22:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.660+0000 7ff492933640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7ff470071e10 0x7ff470074230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.674+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fae2c003990 con 0x7fae3c0a7f80 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.674+0000 7fae32ffd640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fae2006ba30 0x7fae2006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.675+0000 7fae42475640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fae2006ba30 0x7fae2006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.675+0000 7fae42475640 10 monclient: get_auth_request con 0x7fae2006ba30 auth_method 0 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.675+0000 7fae42475640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fae2006ba30 0x7fae2006def0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fae3400e190 tx=0x7fae3400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.675+0000 7fae32ffd640 10 client.4256.objecter ms_handle_connect 0x7fae2006ba30 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fae2c057db0 con 0x7fae3c0a7f80 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 10 client.4256.objecter ms_dispatch 0x7fae3c000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 3 client.4256.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 3 client.4256.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 10 client.4256.objecter checking command 1 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7fae20071e10 0x7fae20074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 20 client.4256.objecter _get_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 15 client.4256.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 20 client.4256.objecter _get_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 20 client.4256.objecter get_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 15 client.4256.objecter _session_command_op_assign 2 1 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 4 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 10 client.4256.objecter _send_command 1 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 --> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae200748e0 con 0x7fae20071e10 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 20 client.4256.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae2c053a60 con 0x7fae3c0a7f80 2023-08-22T23:22:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.678+0000 7fae32ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.679+0000 7fae42c76640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7fae20071e10 0x7fae20074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.679+0000 7fae42c76640 10 monclient: get_auth_request con 0x7fae20071e10 auth_method 0 2023-08-22T23:22:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.682+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f42ec003c60 con 0x7f42f8138710 2023-08-22T23:22:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.682+0000 7f42f4ff9640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f42e406ba30 0x7f42e406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.682+0000 7f42f7fff640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f42e406ba30 0x7f42e406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.682+0000 7f42f7fff640 10 monclient: get_auth_request con 0x7f42e406ba30 auth_method 0 2023-08-22T23:22:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.683+0000 7f42f4ff9640 10 client.4312.objecter ms_handle_connect 0x7f42e406ba30 2023-08-22T23:22:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.683+0000 7f42f7fff640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f42e406ba30 0x7f42e406def0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f42f0007dc0 tx=0x7f42f00072f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.684+0000 7fae32ffd640 10 client.4256.objecter ms_handle_connect 0x7fae20071e10 2023-08-22T23:22:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.684+0000 7fae42c76640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7fae20071e10 0x7fae20074230 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffb6407acf0 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 10 client.4306.objecter ms_dispatch 0x7ffb6c000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 3 client.4306.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 3 client.4306.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 10 client.4306.objecter checking command 1 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7ffb58071dc0 0x7ffb580741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 20 client.4306.objecter _get_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 15 client.4306.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 20 client.4306.objecter _get_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 20 client.4306.objecter get_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 15 client.4306.objecter _session_command_op_assign 1 1 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 4 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 10 client.4306.objecter _send_command 1 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 --> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb58074890 con 0x7ffb58071dc0 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 20 client.4306.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb640769a0 con 0x7ffb6c0aa1a0 2023-08-22T23:22:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.685+0000 7ffb537fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.686+0000 7ffb6b7fe640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7ffb58071dc0 0x7ffb580741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.686+0000 7ffb6b7fe640 10 monclient: get_auth_request con 0x7ffb58071dc0 auth_method 0 2023-08-22T23:22:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7ffb537fe640 10 client.4306.objecter ms_handle_connect 0x7ffb58071dc0 2023-08-22T23:22:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7ffb6b7fe640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7ffb58071dc0 0x7ffb580741e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f42ec057d60 con 0x7f42f8138710 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 10 client.4312.objecter ms_dispatch 0x7f42f8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 3 client.4312.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 3 client.4312.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 10 client.4312.objecter checking command 1 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f42e4071e10 0x7f42e4074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 20 client.4312.objecter _get_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 15 client.4312.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 20 client.4312.objecter _get_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 20 client.4312.objecter get_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 15 client.4312.objecter _session_command_op_assign 4 1 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 4 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 10 client.4312.objecter _send_command 1 2023-08-22T23:22:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 --> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42e40748e0 con 0x7f42e4071e10 2023-08-22T23:22:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 20 client.4312.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42ec0010c0 con 0x7f42f8138710 2023-08-22T23:22:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.690+0000 7f42f4ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.691+0000 7f42f77fe640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f42e4071e10 0x7f42e4074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.691+0000 7f42f77fe640 10 monclient: get_auth_request con 0x7f42e4071e10 auth_method 0 2023-08-22T23:22:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.697+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0bd000da20 con 0x7f0bd81da340 2023-08-22T23:22:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.697+0000 7f0bc6ffd640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0bb806b9e0 0x7f0bb806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.698+0000 7f0bd659c640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0bb806b9e0 0x7f0bb806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.698+0000 7f0bd659c640 10 monclient: get_auth_request con 0x7f0bb806b9e0 auth_method 0 2023-08-22T23:22:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.698+0000 7f0bc6ffd640 10 client.4265.objecter ms_handle_connect 0x7f0bb806b9e0 2023-08-22T23:22:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.698+0000 7f0bd659c640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0bb806b9e0 0x7f0bb806dea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc033000 tx=0x7f0bcc007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.700+0000 7f42f4ff9640 10 client.4312.objecter ms_handle_connect 0x7f42e4071e10 2023-08-22T23:22:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.700+0000 7f42f77fe640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f42e4071e10 0x7f42e4074230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.711+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6ab8003c60 con 0x7f6ac40b2df0 2023-08-22T23:22:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.712+0000 7f6ac17fa640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6ab006ba30 0x7f6ab006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.712+0000 7f6ac897d640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6ab006ba30 0x7f6ab006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.712+0000 7f6ac897d640 10 monclient: get_auth_request con 0x7f6ab006ba30 auth_method 0 2023-08-22T23:22:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.712+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9ca40043a0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.712+0000 7f9ca8ff9640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9c9806b9e0 0x7f9c9806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f9cabfff640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9c9806b9e0 0x7f9c9806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f9cabfff640 10 monclient: get_auth_request con 0x7f9c9806b9e0 auth_method 0 2023-08-22T23:22:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6ab805b3c0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 10 client.4300.objecter ms_dispatch 0x7f6ac4000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 3 client.4300.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 3 client.4300.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac897d640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6ab006ba30 0x7f6ab006def0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40017f0 tx=0x7f6ab4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 10 client.4300.objecter checking command 1 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7f6ab0071e10 0x7f6ab0074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 20 client.4300.objecter _get_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 15 client.4300.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 20 client.4300.objecter _get_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 20 client.4300.objecter get_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 15 client.4300.objecter _session_command_op_assign 3 1 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 4 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 10 client.4300.objecter _send_command 1 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 --> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ab00748e0 con 0x7f6ab0071e10 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 20 client.4300.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 10 client.4300.objecter ms_handle_connect 0x7f6ab006ba30 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ab80539f0 con 0x7f6ac40b2df0 2023-08-22T23:22:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.713+0000 7f6ac17fa640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.714+0000 7f9ca8ff9640 10 client.4262.objecter ms_handle_connect 0x7f9c9806b9e0 2023-08-22T23:22:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.714+0000 7f9cabfff640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9c9806b9e0 0x7f9c9806dea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c036000 tx=0x7f9c9c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.714+0000 7f6ac3fff640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7f6ab0071e10 0x7f6ab0074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.715+0000 7f6ac3fff640 10 monclient: get_auth_request con 0x7f6ab0071e10 auth_method 0 2023-08-22T23:22:32.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0b6610 con 0x7f77f40c6060 2023-08-22T23:22:32.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:32.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 10 monclient: dump: 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 10 monclient: _finish_auth 0 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f77f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 5 monclient: authenticate success, global_id 4268 2023-08-22T23:22:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 10 monclient: _renew_subs 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f40c5b00 con 0x7f77f40c6060 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 10 monclient: _renew_subs 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f40c10c0 con 0x7f77f40c6060 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.716+0000 7f7802336640 1 librados: init done 2023-08-22T23:22:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.717+0000 7f7802336640 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.717+0000 7f7802336640 15 client.4268.objecter _session_command_op_assign -1 1 2023-08-22T23:22:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.717+0000 7f7802336640 20 client.4268.objecter _get_session osd=-1 returning homeless 2023-08-22T23:22:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.717+0000 7f7802336640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.717+0000 7f7802336640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.717+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77f4033570 con 0x7f77f40c6060 2023-08-22T23:22:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.719+0000 7f6ac17fa640 10 client.4300.objecter ms_handle_connect 0x7f6ab0071e10 2023-08-22T23:22:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.719+0000 7f6ac3fff640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7f6ab0071e10 0x7f6ab0074230 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.722+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== osd.7 v2:172.21.15.52:6810/3454580439 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff4700748e0 con 0x7ff470071e10 2023-08-22T23:22:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.722+0000 7ff482ffd640 10 client.4305.objecter ms_dispatch 0x7ff48c000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.722+0000 7ff482ffd640 10 client.4305.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.722+0000 7ff482ffd640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 4 2023-08-22T23:22:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.722+0000 7ff482ffd640 15 client.4305.objecter _session_command_op_remove 7 1 2023-08-22T23:22:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0bd0026390 con 0x7f0bd81da340 2023-08-22T23:22:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 10 client.4265.objecter ms_dispatch 0x7f0bd8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 3 client.4265.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 3 client.4265.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 10 client.4265.objecter checking command 1 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f0bb8071dc0 0x7f0bb80741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 20 client.4265.objecter _get_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 15 client.4265.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 20 client.4265.objecter _get_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 20 client.4265.objecter get_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 15 client.4265.objecter _session_command_op_assign 5 1 2023-08-22T23:22:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 4 2023-08-22T23:22:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 10 client.4265.objecter _send_command 1 2023-08-22T23:22:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 --> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bb8074890 con 0x7f0bb8071dc0 2023-08-22T23:22:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 20 client.4265.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bd0054580 con 0x7f0bd81da340 2023-08-22T23:22:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f0bc6ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc0b6770 con 0x7f77f40c6060 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0b6ae0 con 0x7f77f40c6060 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f1ffb640 10 monclient: dump: 2023-08-22T23:22:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:32.854 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.732+0000 7f77f37fe640 4 set_mon_vals no callback set 2023-08-22T23:22:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.733+0000 7f0bd5d9b640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f0bb8071dc0 0x7f0bb80741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.733+0000 7f0bd5d9b640 10 monclient: get_auth_request con 0x7f0bb8071dc0 auth_method 0 2023-08-22T23:22:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.737+0000 7f0bc6ffd640 10 client.4265.objecter ms_handle_connect 0x7f0bb8071dc0 2023-08-22T23:22:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.737+0000 7f0bd5d9b640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f0bb8071dc0 0x7f0bb80741e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 10 client.4305.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 15 client.4305.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 20 client.4305.objecter _get_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 20 client.4305.objecter _get_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 15 client.4305.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 20 client.4305.objecter get_session s=0x7ff470071740 osd=7 3 2023-08-22T23:22:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 15 client.4305.objecter _session_command_op_assign 7 2 2023-08-22T23:22:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 4 2023-08-22T23:22:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 10 client.4305.objecter _send_command 2 2023-08-22T23:22:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.740+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 --> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff48c0ac3f0 con 0x7ff470071e10 2023-08-22T23:22:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9ca4012070 con 0x7f9cac0adfc0 2023-08-22T23:22:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 10 client.4262.objecter ms_dispatch 0x7f9cac000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 3 client.4262.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 3 client.4262.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 10 client.4262.objecter checking command 1 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f9c98071dc0 0x7f9c980741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 20 client.4262.objecter _get_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 15 client.4262.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 20 client.4262.objecter _get_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 20 client.4262.objecter get_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 15 client.4262.objecter _session_command_op_assign 6 1 2023-08-22T23:22:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 4 2023-08-22T23:22:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 10 client.4262.objecter _send_command 1 2023-08-22T23:22:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c98074890 con 0x7f9c98071dc0 2023-08-22T23:22:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.752+0000 7f9ca8ff9640 20 client.4262.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.753+0000 7f9cab7fe640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f9c98071dc0 0x7f9c980741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.753+0000 7f9cab7fe640 10 monclient: get_auth_request con 0x7f9c98071dc0 auth_method 0 2023-08-22T23:22:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.754+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f77fc0b4040 con 0x7f77f40c6060 2023-08-22T23:22:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.754+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ca4054bd0 con 0x7f9cac0adfc0 2023-08-22T23:22:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.754+0000 7f9ca8ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.754+0000 7f77f1ffb640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f77e006ba10 0x7f77e006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.754+0000 7f7801334640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f77e006ba10 0x7f77e006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.754+0000 7f7801334640 10 monclient: get_auth_request con 0x7f77e006ba10 auth_method 0 2023-08-22T23:22:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.755+0000 7f7801334640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f77e006ba10 0x7f77e006ded0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006f40 tx=0x7f77f8009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.755+0000 7f77f1ffb640 10 client.4268.objecter ms_handle_connect 0x7f77e006ba10 2023-08-22T23:22:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== osd.2 v2:172.21.15.29:6800/1029571786 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fae200748e0 con 0x7fae20071e10 2023-08-22T23:22:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7fae32ffd640 10 client.4256.objecter ms_dispatch 0x7fae3c000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7fae32ffd640 10 client.4256.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7fae32ffd640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 4 2023-08-22T23:22:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7fae32ffd640 15 client.4256.objecter _session_command_op_remove 2 1 2023-08-22T23:22:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7f9ca8ff9640 10 client.4262.objecter ms_handle_connect 0x7f9c98071dc0 2023-08-22T23:22:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.756+0000 7f9cab7fe640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f9c98071dc0 0x7f9c980741e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f77fc0f3710 con 0x7f77f40c6060 2023-08-22T23:22:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 10 client.4268.objecter ms_dispatch 0x7f77fc000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 3 client.4268.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 3 client.4268.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 10 client.4268.objecter checking command 1 2023-08-22T23:22:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f77e0071df0 0x7f77e0074210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 20 client.4268.objecter _get_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 15 client.4268.objecter _session_command_op_remove -1 1 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 20 client.4268.objecter _get_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 20 client.4268.objecter get_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 15 client.4268.objecter _session_command_op_assign 0 1 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 4 2023-08-22T23:22:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 10 client.4268.objecter _send_command 1 2023-08-22T23:22:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 --> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77e00748c0 con 0x7f77e0071df0 2023-08-22T23:22:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 20 client.4268.objecter dump_active .. 0 homeless 2023-08-22T23:22:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77fc0ef350 con 0x7f77f40c6060 2023-08-22T23:22:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.759+0000 7f77f1ffb640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:22:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== osd.1 v2:172.21.15.29:6824/2619884743 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ffb58074890 con 0x7ffb58071dc0 2023-08-22T23:22:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7ffb537fe640 10 client.4306.objecter ms_dispatch 0x7ffb6c000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7ffb537fe640 10 client.4306.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7ffb537fe640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 4 2023-08-22T23:22:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7ffb537fe640 15 client.4306.objecter _session_command_op_remove 1 1 2023-08-22T23:22:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7f7800b33640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f77e0071df0 0x7f77e0074210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.760+0000 7f7800b33640 10 monclient: get_auth_request con 0x7f77e0071df0 auth_method 0 2023-08-22T23:22:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff482ffd640 1 -- 172.21.15.29:0/235105092 <== osd.7 v2:172.21.15.52:6810/3454580439 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff484006b50 con 0x7ff470071e10 2023-08-22T23:22:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff482ffd640 10 client.4305.objecter ms_dispatch 0x7ff48c000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff482ffd640 10 client.4305.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff482ffd640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 4 2023-08-22T23:22:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff482ffd640 15 client.4305.objecter _session_command_op_remove 7 2 2023-08-22T23:22:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff494396640 10 librados: watch_flush enter 2023-08-22T23:22:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.763+0000 7ff494396640 10 librados: watch_flush exit 2023-08-22T23:22:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 10 client.4305.objecter close_session for osd.7 2023-08-22T23:22:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7ff470071e10 msgr2=0x7ff470074230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7ff470071e10 0x7ff470074230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 20 client.4305.objecter put_session s=0x7ff470071740 osd=7 1 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 20 client.4305.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 10 client.4305.objecter successfully canceled tick 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff47006ba30 msgr2=0x7ff47006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff47006ba30 0x7ff47006def0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff488004f60 tx=0x7ff488007e30 comp rx=0 tx=0).stop 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 10 monclient: shutdown 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.764+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0aafd0 msgr2=0x7ff48c06e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.765+0000 7ff494396640 1 --2- 172.21.15.29:0/235105092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0aafd0 0x7ff48c06e520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff47c000c00 tx=0x7ff47c0046c0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.765+0000 7ff492933640 1 -- 172.21.15.29:0/235105092 reap_dead start 2023-08-22T23:22:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.765+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 shutdown_connections 2023-08-22T23:22:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.765+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 >> 172.21.15.29:0/235105092 conn(0x7ff48c0b27e0 msgr2=0x7ff48c0af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.767+0000 7f7800b33640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f77e0071df0 0x7f77e0074210 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.767+0000 7f77f1ffb640 10 client.4268.objecter ms_handle_connect 0x7f77e0071df0 2023-08-22T23:22:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.767+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 shutdown_connections 2023-08-22T23:22:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.767+0000 7ff494396640 1 -- 172.21.15.29:0/235105092 wait complete. 2023-08-22T23:22:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.768+0000 7ff494396640 1 librados: shutdown 2023-08-22T23:22:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.768+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== osd.4 v2:172.21.15.52:6802/2157117708 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f42e40748e0 con 0x7f42e4071e10 2023-08-22T23:22:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.768+0000 7f42f4ff9640 10 client.4312.objecter ms_dispatch 0x7f42f8000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.768+0000 7f42f4ff9640 10 client.4312.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.768+0000 7f42f4ff9640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 4 2023-08-22T23:22:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.768+0000 7f42f4ff9640 15 client.4312.objecter _session_command_op_remove 4 1 2023-08-22T23:22:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 15 client.4256.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 20 client.4256.objecter _get_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 20 client.4256.objecter _get_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 15 client.4256.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== osd.5 v2:172.21.15.52:6826/4164832567 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f0bb8074890 con 0x7f0bb8071dc0 2023-08-22T23:22:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7f0bc6ffd640 10 client.4265.objecter ms_dispatch 0x7f0bd8000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7f0bc6ffd640 10 client.4265.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 20 client.4256.objecter get_session s=0x7fae20071740 osd=2 3 2023-08-22T23:22:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 15 client.4256.objecter _session_command_op_assign 2 2 2023-08-22T23:22:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7f0bc6ffd640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 4 2023-08-22T23:22:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7f0bc6ffd640 15 client.4265.objecter _session_command_op_remove 5 1 2023-08-22T23:22:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 4 2023-08-22T23:22:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 10 client.4256.objecter _send_command 2 2023-08-22T23:22:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.785+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 --> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae3c145260 con 0x7fae20071e10 2023-08-22T23:22:32.877 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-08-22T23:22:32.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-08-22T23:22:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 10 client.4306.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 15 client.4306.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 20 client.4306.objecter _get_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 20 client.4306.objecter _get_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 15 client.4306.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 20 client.4306.objecter get_session s=0x7ffb580716f0 osd=1 3 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 15 client.4306.objecter _session_command_op_assign 1 2 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 4 2023-08-22T23:22:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 10 client.4306.objecter _send_command 2 2023-08-22T23:22:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.794+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 --> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffb6c0ac3f0 con 0x7ffb58071dc0 2023-08-22T23:22:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.795+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== osd.3 v2:172.21.15.29:6815/4227673673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f6ab00748e0 con 0x7f6ab0071e10 2023-08-22T23:22:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.795+0000 7f6ac17fa640 10 client.4300.objecter ms_dispatch 0x7f6ac4000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.795+0000 7f6ac17fa640 10 client.4300.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.795+0000 7f6ac17fa640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 4 2023-08-22T23:22:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.795+0000 7f6ac17fa640 15 client.4300.objecter _session_command_op_remove 3 1 2023-08-22T23:22:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.799+0000 7fae32ffd640 1 -- 172.21.15.29:0/1832659047 <== osd.2 v2:172.21.15.29:6800/1029571786 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae3c145260 con 0x7fae20071e10 2023-08-22T23:22:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.799+0000 7fae32ffd640 10 client.4256.objecter ms_dispatch 0x7fae3c000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.799+0000 7fae32ffd640 10 client.4256.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.799+0000 7fae32ffd640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 4 2023-08-22T23:22:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.799+0000 7fae32ffd640 15 client.4256.objecter _session_command_op_remove 2 2 2023-08-22T23:22:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 10 librados: watch_flush enter 2023-08-22T23:22:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 10 librados: watch_flush exit 2023-08-22T23:22:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 10 client.4256.objecter close_session for osd.2 2023-08-22T23:22:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7fae20071e10 msgr2=0x7fae20074230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7fae20071e10 0x7fae20074230 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 20 client.4256.objecter put_session s=0x7fae20071740 osd=2 1 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 20 client.4256.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 10 client.4256.objecter successfully canceled tick 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fae2006ba30 msgr2=0x7fae2006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fae2006ba30 0x7fae2006def0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fae3400e190 tx=0x7fae3400b040 comp rx=0 tx=0).stop 2023-08-22T23:22:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 10 monclient: shutdown 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0a7f80 msgr2=0x7fae3c142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae43477640 1 --2- 172.21.15.29:0/1832659047 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fae3c0a7f80 0x7fae3c142200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c400 tx=0x7fae2c00c8c0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.800+0000 7fae42c76640 1 -- 172.21.15.29:0/1832659047 reap_dead start 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.802+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 shutdown_connections 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.802+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 >> 172.21.15.29:0/1832659047 conn(0x7fae3c0b67d0 msgr2=0x7fae3c0ae340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.802+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 shutdown_connections 2023-08-22T23:22:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.802+0000 7fae43477640 1 -- 172.21.15.29:0/1832659047 wait complete. 2023-08-22T23:22:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.802+0000 7fae43477640 1 librados: shutdown 2023-08-22T23:22:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.802+0000 7f42fe6c5640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 15 client.4312.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 20 client.4312.objecter _get_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 20 client.4312.objecter _get_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 15 client.4312.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 20 client.4312.objecter get_session s=0x7f42e4071740 osd=4 3 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 15 client.4312.objecter _session_command_op_assign 4 2 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 4 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 10 client.4312.objecter _send_command 2 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.803+0000 7f42fe6c5640 1 -- 172.21.15.29:0/464047534 --> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f42f813afa0 con 0x7f42e4071e10 2023-08-22T23:22:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb537fe640 1 -- 172.21.15.29:0/868261135 <== osd.1 v2:172.21.15.29:6824/2619884743 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffb5c0058b0 con 0x7ffb58071dc0 2023-08-22T23:22:32.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb537fe640 10 client.4306.objecter ms_dispatch 0x7ffb6c000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb537fe640 10 client.4306.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb537fe640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 4 2023-08-22T23:22:32.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb537fe640 15 client.4306.objecter _session_command_op_remove 1 2 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb7135d640 10 librados: watch_flush enter 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb7135d640 10 librados: watch_flush exit 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb7135d640 10 client.4306.objecter close_session for osd.1 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.804+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7ffb58071dc0 msgr2=0x7ffb580741e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.805+0000 7ffb7135d640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7ffb58071dc0 0x7ffb580741e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 20 client.4306.objecter put_session s=0x7ffb580716f0 osd=1 1 2023-08-22T23:22:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 20 client.4306.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 10 client.4306.objecter successfully canceled tick 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffb5806b9e0 msgr2=0x7ffb5806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffb5806b9e0 0x7ffb5806dea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffb600010e0 tx=0x7ffb600078f0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 10 monclient: shutdown 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb6c0aa1a0 msgr2=0x7ffb6c1c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 1 --2- 172.21.15.29:0/868261135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb6c0aa1a0 0x7ffb6c1c8e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb64000f80 tx=0x7ffb64004ab0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb6b7fe640 1 -- 172.21.15.29:0/868261135 reap_dead start 2023-08-22T23:22:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 shutdown_connections 2023-08-22T23:22:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.806+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 >> 172.21.15.29:0/868261135 conn(0x7ffb6c0b27e0 msgr2=0x7ffb6c1d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.807+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 shutdown_connections 2023-08-22T23:22:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.807+0000 7ffb7135d640 1 -- 172.21.15.29:0/868261135 wait complete. 2023-08-22T23:22:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.807+0000 7ffb7135d640 1 librados: shutdown 2023-08-22T23:22:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f42f4ff9640 1 -- 172.21.15.29:0/464047534 <== osd.4 v2:172.21.15.52:6802/2157117708 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f42f813afa0 con 0x7f42e4071e10 2023-08-22T23:22:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f42f4ff9640 10 client.4312.objecter ms_dispatch 0x7f42f8000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f42f4ff9640 10 client.4312.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f42f4ff9640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 4 2023-08-22T23:22:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f42f4ff9640 15 client.4312.objecter _session_command_op_remove 4 2 2023-08-22T23:22:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== osd.6 v2:172.21.15.52:6818/2556931765 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f9c98074890 con 0x7f9c98071dc0 2023-08-22T23:22:32.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f9ca8ff9640 10 client.4262.objecter ms_dispatch 0x7f9cac000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f9ca8ff9640 10 client.4262.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f9ca8ff9640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 4 2023-08-22T23:22:32.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.809+0000 7f9ca8ff9640 15 client.4262.objecter _session_command_op_remove 6 1 2023-08-22T23:22:32.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.811+0000 7f0bdc8aa640 10 client.4265.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 15 client.4265.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 20 client.4265.objecter _get_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 20 client.4265.objecter _get_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 15 client.4265.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 20 client.4265.objecter get_session s=0x7f0bb80716f0 osd=5 3 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 15 client.4265.objecter _session_command_op_assign 5 2 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 4 2023-08-22T23:22:32.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 10 client.4265.objecter _send_command 2 2023-08-22T23:22:32.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.812+0000 7f0bdc8aa640 1 -- 172.21.15.29:0/4122970911 --> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0bd80ae530 con 0x7f0bb8071dc0 2023-08-22T23:22:32.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 10 librados: watch_flush enter 2023-08-22T23:22:32.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 10 librados: watch_flush exit 2023-08-22T23:22:32.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 10 client.4312.objecter close_session for osd.4 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f42e4071e10 msgr2=0x7f42e4074230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f42e4071e10 0x7f42e4074230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 20 client.4312.objecter put_session s=0x7f42e4071740 osd=4 1 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 20 client.4312.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.816+0000 7f42de7fc640 10 client.4312.objecter successfully canceled tick 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f42e406ba30 msgr2=0x7f42e406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f42e406ba30 0x7f42e406def0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f42f0007dc0 tx=0x7f42f00072f0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 10 monclient: shutdown 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42f8138710 msgr2=0x7f42f813a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 --2- 172.21.15.29:0/464047534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42f8138710 0x7f42f813a630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c390 tx=0x7f42ec00c850 comp rx=0 tx=0).stop 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42fcc62640 1 -- 172.21.15.29:0/464047534 reap_dead start 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 shutdown_connections 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 >> 172.21.15.29:0/464047534 conn(0x7f42f80aee40 msgr2=0x7f42f8136df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.817+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 shutdown_connections 2023-08-22T23:22:32.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f42de7fc640 1 -- 172.21.15.29:0/464047534 wait complete. 2023-08-22T23:22:32.898 INFO:teuthology.orchestra.run.smithi029.stdout:68719476741 2023-08-22T23:22:32.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-08-22T23:22:32.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f42de7fc640 1 librados: shutdown 2023-08-22T23:22:32.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f0bc6ffd640 1 -- 172.21.15.29:0/4122970911 <== osd.5 v2:172.21.15.52:6826/4164832567 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0bc8009c50 con 0x7f0bb8071dc0 2023-08-22T23:22:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f0bc6ffd640 10 client.4265.objecter ms_dispatch 0x7f0bd8000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f0bc6ffd640 10 client.4265.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f0bc6ffd640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 4 2023-08-22T23:22:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.818+0000 7f0bc6ffd640 15 client.4265.objecter _session_command_op_remove 5 2 2023-08-22T23:22:32.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 10 librados: watch_flush enter 2023-08-22T23:22:32.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 10 librados: watch_flush exit 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 10 client.4265.objecter close_session for osd.5 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f0bb8071dc0 msgr2=0x7f0bb80741e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f0bb8071dc0 0x7f0bb80741e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 20 client.4265.objecter put_session s=0x7f0bb80716f0 osd=5 1 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 20 client.4265.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 10 client.4265.objecter successfully canceled tick 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0bb806b9e0 msgr2=0x7f0bb806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0bb806b9e0 0x7f0bb806dea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc033000 tx=0x7f0bcc007000 comp rx=0 tx=0).stop 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 10 monclient: shutdown 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bd81da340 msgr2=0x7f0bd81da7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bc4ff9640 1 --2- 172.21.15.29:0/4122970911 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0bd81da340 0x7f0bd81da7c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80b34f0 tx=0x7f0bd000fb40 comp rx=0 tx=0).stop 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.819+0000 7f0bd6d9d640 1 -- 172.21.15.29:0/4122970911 reap_dead start 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.820+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 shutdown_connections 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.820+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 >> 172.21.15.29:0/4122970911 conn(0x7f0bd80afad0 msgr2=0x7f0bd8141990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 shutdown_connections 2023-08-22T23:22:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f0bc4ff9640 1 -- 172.21.15.29:0/4122970911 wait complete. 2023-08-22T23:22:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f0bc4ff9640 1 librados: shutdown 2023-08-22T23:22:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 15 client.4300.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 20 client.4300.objecter _get_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 20 client.4300.objecter _get_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 15 client.4300.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 20 client.4300.objecter get_session s=0x7f6ab0071740 osd=3 3 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 15 client.4300.objecter _session_command_op_assign 3 2 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 4 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 10 client.4300.objecter _send_command 2 2023-08-22T23:22:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.821+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 --> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ac40aa8d0 con 0x7f6ab0071e10 2023-08-22T23:22:32.906 INFO:teuthology.orchestra.run.smithi029.stdout:68719476741 2023-08-22T23:22:32.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-08-22T23:22:32.907 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-08-22T23:22:32.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-08-22T23:22:32.908 INFO:teuthology.orchestra.run.smithi029.stdout:64424509446 2023-08-22T23:22:32.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-08-22T23:22:32.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 10 client.4262.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 15 client.4262.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 20 client.4262.objecter _get_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 20 client.4262.objecter _get_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 15 client.4262.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 20 client.4262.objecter get_session s=0x7f9c980716f0 osd=6 3 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 15 client.4262.objecter _session_command_op_assign 6 2 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 4 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 10 client.4262.objecter _send_command 2 2023-08-22T23:22:32.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.838+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9cac0a6030 con 0x7f9c98071dc0 2023-08-22T23:22:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.842+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== osd.0 v2:172.21.15.29:6808/1473840652 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f77e00748c0 con 0x7f77e0071df0 2023-08-22T23:22:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.842+0000 7f77f1ffb640 10 client.4268.objecter ms_dispatch 0x7f77fc000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:22:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.842+0000 7f77f1ffb640 10 client.4268.objecter _finish_command 1 = system:0 2023-08-22T23:22:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.842+0000 7f77f1ffb640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 4 2023-08-22T23:22:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.842+0000 7f77f1ffb640 15 client.4268.objecter _session_command_op_remove 0 1 2023-08-22T23:22:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9ca8ff9640 1 -- 172.21.15.29:0/2957632360 <== osd.6 v2:172.21.15.52:6818/2556931765 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9ca0001440 con 0x7f9c98071dc0 2023-08-22T23:22:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9ca8ff9640 10 client.4262.objecter ms_dispatch 0x7f9cac000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9ca8ff9640 10 client.4262.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9ca8ff9640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 4 2023-08-22T23:22:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9ca8ff9640 15 client.4262.objecter _session_command_op_remove 6 2 2023-08-22T23:22:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 10 librados: watch_flush enter 2023-08-22T23:22:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 10 librados: watch_flush exit 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 10 client.4262.objecter close_session for osd.6 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f9c98071dc0 msgr2=0x7f9c980741e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f9c98071dc0 0x7f9c980741e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 20 client.4262.objecter put_session s=0x7f9c980716f0 osd=6 1 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 20 client.4262.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 10 client.4262.objecter successfully canceled tick 2023-08-22T23:22:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9c9806b9e0 msgr2=0x7f9c9806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9c9806b9e0 0x7f9c9806dea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c036000 tx=0x7f9c9c03d040 comp rx=0 tx=0).stop 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 10 monclient: shutdown 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0adfc0 msgr2=0x7f9cac1da4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 --2- 172.21.15.29:0/2957632360 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9cac0adfc0 0x7f9cac1da4f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400cd70 tx=0x7f9ca400a660 comp rx=0 tx=0).stop 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb0df4640 1 -- 172.21.15.29:0/2957632360 reap_dead start 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 shutdown_connections 2023-08-22T23:22:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.844+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 >> 172.21.15.29:0/2957632360 conn(0x7f9cac0aee40 msgr2=0x7f9cac1d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.845+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 shutdown_connections 2023-08-22T23:22:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.845+0000 7f9cb2857640 1 -- 172.21.15.29:0/2957632360 wait complete. 2023-08-22T23:22:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.845+0000 7f9cb2857640 1 librados: shutdown 2023-08-22T23:22:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.853+0000 7f6ac17fa640 1 -- 172.21.15.29:0/4192804943 <== osd.3 v2:172.21.15.29:6815/4227673673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6abc0344d0 con 0x7f6ab0071e10 2023-08-22T23:22:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.853+0000 7f6ac17fa640 10 client.4300.objecter ms_dispatch 0x7f6ac4000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.853+0000 7f6ac17fa640 10 client.4300.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.853+0000 7f6ac17fa640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 4 2023-08-22T23:22:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.853+0000 7f6ac17fa640 15 client.4300.objecter _session_command_op_remove 3 2 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 10 librados: watch_flush enter 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 10 librados: watch_flush exit 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 10 client.4300.objecter close_session for osd.3 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7f6ab0071e10 msgr2=0x7f6ab0074230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7f6ab0071e10 0x7f6ab0074230 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 20 client.4300.objecter put_session s=0x7f6ab0071740 osd=3 1 2023-08-22T23:22:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 20 client.4300.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 10 client.4300.objecter successfully canceled tick 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6ab006ba30 msgr2=0x7f6ab006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6ab006ba30 0x7f6ab006def0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40017f0 tx=0x7f6ab4009040 comp rx=0 tx=0).stop 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 10 monclient: shutdown 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ac40b2df0 msgr2=0x7f6ac41d9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 --2- 172.21.15.29:0/4192804943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ac40b2df0 0x7f6ac41d9470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c390 tx=0x7f6ab800c850 comp rx=0 tx=0).stop 2023-08-22T23:22:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac917e640 1 -- 172.21.15.29:0/4192804943 reap_dead start 2023-08-22T23:22:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 shutdown_connections 2023-08-22T23:22:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 >> 172.21.15.29:0/4192804943 conn(0x7f6ac40b29e0 msgr2=0x7f6ac4073190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 shutdown_connections 2023-08-22T23:22:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 -- 172.21.15.29:0/4192804943 wait complete. 2023-08-22T23:22:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.854+0000 7f6ac997f640 1 librados: shutdown 2023-08-22T23:22:32.921 INFO:teuthology.orchestra.run.smithi029.stdout:64424509447 2023-08-22T23:22:32.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 15 client.4268.objecter _session_command_op_assign -1 2 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 20 client.4268.objecter _get_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 20 client.4268.objecter _get_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 15 client.4268.objecter _session_command_op_remove -1 2 2023-08-22T23:22:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 20 client.4268.objecter get_session s=0x7f77e0071720 osd=0 3 2023-08-22T23:22:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 15 client.4268.objecter _session_command_op_assign 0 2 2023-08-22T23:22:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 4 2023-08-22T23:22:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 10 client.4268.objecter _send_command 2 2023-08-22T23:22:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.863+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 --> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f77f40c7cd0 con 0x7f77e0071df0 2023-08-22T23:22:32.924 INFO:teuthology.orchestra.run.smithi029.stdout:68719476742 2023-08-22T23:22:32.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-08-22T23:22:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.872+0000 7f77f1ffb640 1 -- 172.21.15.29:0/1192927987 <== osd.0 v2:172.21.15.29:6808/1473840652 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f77f40c7cd0 con 0x7f77e0071df0 2023-08-22T23:22:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.872+0000 7f77f1ffb640 10 client.4268.objecter ms_dispatch 0x7f77fc000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:22:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.872+0000 7f77f1ffb640 10 client.4268.objecter _finish_command 2 = system:0 2023-08-22T23:22:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.872+0000 7f77f1ffb640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 4 2023-08-22T23:22:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.872+0000 7f77f1ffb640 15 client.4268.objecter _session_command_op_remove 0 2 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 10 librados: watch_flush enter 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 10 librados: watch_flush exit 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 10 client.4268.objecter close_session for osd.0 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f77e0071df0 msgr2=0x7f77e0074210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f77e0071df0 0x7f77e0074210 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 20 client.4268.objecter put_session s=0x7f77e0071720 osd=0 1 2023-08-22T23:22:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 20 client.4268.objecter shutdown clearing up homeless session... 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 10 client.4268.objecter successfully canceled tick 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f77e006ba10 msgr2=0x7f77e006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f77e006ba10 0x7f77e006ded0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006f40 tx=0x7f77f8009240 comp rx=0 tx=0).stop 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 10 monclient: shutdown 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f77f40c6060 msgr2=0x7f77f40c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 --2- 172.21.15.29:0/1192927987 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f77f40c6060 0x7f77f40c55c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77fc04f3b0 tx=0x7f77fc072a70 comp rx=0 tx=0).stop 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7801b35640 1 -- 172.21.15.29:0/1192927987 reap_dead start 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 shutdown_connections 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 >> 172.21.15.29:0/1192927987 conn(0x7f77f40060a0 msgr2=0x7f77f4029240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 shutdown_connections 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 -- 172.21.15.29:0/1192927987 wait complete. 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:32.873+0000 7f7802336640 1 librados: shutdown 2023-08-22T23:22:32.929 INFO:teuthology.orchestra.run.smithi029.stdout:68719476741 2023-08-22T23:22:32.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-08-22T23:22:33.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.127+0000 7f9772807640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.127+0000 7f9772807640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.127+0000 7f9772807640 10 monclient: monmap: 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.129261+0000 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.129261+0000 2023-08-22T23:22:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.131 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.128+0000 7fedac2b7640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.128+0000 7fedac2b7640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.128+0000 7fedac2b7640 10 monclient: monmap: 2023-08-22T23:22:33.132 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.129873+0000 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.129873+0000 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.133 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.134 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.129+0000 7f0ccdf29640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.129+0000 7f0ccdf29640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.129+0000 7f0ccdf29640 10 monclient: monmap: 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.130938+0000 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.130938+0000 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.136 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.136 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7f9772807640 1 Processor -- start 2023-08-22T23:22:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7f9772807640 1 -- start start 2023-08-22T23:22:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7f9772807640 10 monclient: init 2023-08-22T23:22:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7f0ccdf29640 1 Processor -- start 2023-08-22T23:22:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f0ccdf29640 1 -- start start 2023-08-22T23:22:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f0ccdf29640 10 monclient: init 2023-08-22T23:22:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-08-22T23:22:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): picked mon.noname-b con 0x7f976c0adfc0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f976c0713c0 0x7f976c0a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): picked mon.noname-c con 0x7f976c0713c0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f976c0a8a10 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976c0a6450 con 0x7f976c0a8a10 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976c0ae3a0 con 0x7f976c0713c0 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0a8250 con 0x7f976c0adfc0 2023-08-22T23:22:33.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.131+0000 7f9772807640 10 monclient(hunting): authenticate will time out at 1517.382297s 2023-08-22T23:22:33.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7fedac2b7640 1 Processor -- start 2023-08-22T23:22:33.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7fedac2b7640 1 -- start start 2023-08-22T23:22:33.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.130+0000 7fedac2b7640 10 monclient: init 2023-08-22T23:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-08-22T23:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cc8070350 0x7f0cc813c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc8070350 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:33.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f976bfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0cc8070730 0x7f0cc813ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc8070730 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:33.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0cc813f460 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:33.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-08-22T23:22:33.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feda40aa1a0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:33.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7feda40aafd0 0x7feda40ae380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): picked mon.noname-c con 0x7feda40aafd0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:33.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feda4149380 0x7feda41500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): picked mon.noname-a con 0x7feda4149380 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40a6010 con 0x7feda40aa1a0 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda40aa580 con 0x7feda4149380 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda40a5d00 con 0x7feda40aafd0 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7fedac2b7640 10 monclient(hunting): authenticate will time out at 1517.383279s 2023-08-22T23:22:33.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc806f820 con 0x7f0cc813f460 2023-08-22T23:22:33.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc80ac3a0 con 0x7f0cc8070350 2023-08-22T23:22:33.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc806fa70 con 0x7f0cc8070730 2023-08-22T23:22:33.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.132+0000 7f0ccdf29640 10 monclient(hunting): authenticate will time out at 1517.383413s 2023-08-22T23:22:33.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f976bfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43444/0 (socket says 172.21.15.29:43444) 2023-08-22T23:22:33.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f976bfff640 1 -- 172.21.15.29:0/503279459 learned_addr learned my addr 172.21.15.29:0/503279459 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f976bfff640 10 monclient(hunting): get_auth_request con 0x7f976c0adfc0 auth_method 0 2023-08-22T23:22:33.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f976bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f976bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f976bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f0feed19640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f0feed19640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7f0feed19640 10 monclient: monmap: 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.134792+0000 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.134792+0000 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.148 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.149 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.149 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.149 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.149 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.133+0000 7feda9852640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7feda40aafd0 0x7feda40ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7feda9852640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7feda40aafd0 0x7feda40ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39638/0 (socket says 172.21.15.29:39638) 2023-08-22T23:22:33.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7feda9852640 1 -- 172.21.15.29:0/1985103087 learned_addr learned my addr 172.21.15.29:0/1985103087 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7feda9852640 10 monclient(hunting): get_auth_request con 0x7feda40aafd0 auth_method 0 2023-08-22T23:22:33.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7feda9852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7feda9852640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7feda9852640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7fe1cc483640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7fe1cc483640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7fe1cc483640 10 monclient: monmap: 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.136104+0000 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.136104+0000 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.152 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.153 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.153 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.153 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.153 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.153 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.153 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.134+0000 7f0feed19640 1 Processor -- start 2023-08-22T23:22:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7f0feed19640 1 -- start start 2023-08-22T23:22:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7f0feed19640 10 monclient: init 2023-08-22T23:22:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7feda9852640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7feda9852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7feda9852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7f976b7fe640 1 --2- 172.21.15.29:0/503279459 >> v2:172.21.15.52:3300/0 conn(0x7f976c0713c0 0x7f976c0a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7f0cbffff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0cc8070730 0x7f0cc813ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7fe1cc483640 1 Processor -- start 2023-08-22T23:22:33.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-08-22T23:22:33.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fe0038ef0 0x7f0fe00c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe0038ef0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:33.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 0x7f0fe00c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe00392d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe00c8310 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe00374a0 con 0x7f0fe00c8310 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe00372c0 con 0x7f0fe0038ef0 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0037780 con 0x7f0fe00392d0 2023-08-22T23:22:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7f0feed19640 10 monclient(hunting): authenticate will time out at 1517.386596s 2023-08-22T23:22:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7fe1cc483640 1 -- start start 2023-08-22T23:22:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.135+0000 7fe1cc483640 10 monclient: init 2023-08-22T23:22:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7fe1cc483640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7fe1cc483640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7fe1cc483640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1c40acba0 0x7fe1c4141120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7fe1cc483640 10 monclient(hunting): picked mon.noname-a con 0x7fe1c40acba0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.136+0000 7fe1cc483640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c406b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 10 monclient(hunting): picked mon.noname-b con 0x7fe1c406f150 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1c40aeef0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40ad3f0 con 0x7fe1c40aeef0 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c4071610 con 0x7fe1c40acba0 2023-08-22T23:22:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40705b0 con 0x7fe1c406f150 2023-08-22T23:22:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7fe1cc483640 10 monclient(hunting): authenticate will time out at 1517.387529s 2023-08-22T23:22:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2023-08-22T23:22:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 1 -- 172.21.15.29:0/503279459 >> v2:172.21.15.52:3300/0 conn(0x7f976c0713c0 msgr2=0x7f976c0a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 1 --2- 172.21.15.29:0/503279459 >> v2:172.21.15.52:3300/0 conn(0x7f976c0713c0 0x7f976c0a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 1 -- 172.21.15.29:0/503279459 >> v1:172.21.15.29:6789/0 conn(0x7f976c0a8a10 legacy=0x7f976c13c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 1 monclient: found mon.noname-b 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976bfff640 1 -- 172.21.15.29:0/503279459 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c0a84f0 con 0x7f976c0adfc0 2023-08-22T23:22:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976b7fe640 10 monclient: get_auth_request con 0x7f976c0713c0 auth_method 0 2023-08-22T23:22:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f976b7fe640 1 --2- 172.21.15.29:0/503279459 >> v2:172.21.15.52:3300/0 conn(0x7f976c0713c0 0x7f976c0a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-22T23:22:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f0cbffff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0cc8070730 0x7f0cc813ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39652/0 (socket says 172.21.15.29:39652) 2023-08-22T23:22:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f0cbffff640 1 -- 172.21.15.29:0/4237789787 learned_addr learned my addr 172.21.15.29:0/4237789787 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f0cbffff640 10 monclient(hunting): get_auth_request con 0x7f0cc8070730 auth_method 0 2023-08-22T23:22:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f0cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f0cbffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7f0cbffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2023-08-22T23:22:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 1 -- 172.21.15.29:0/1985103087 >> v2:172.21.15.29:3300/0 conn(0x7feda4149380 msgr2=0x7feda41500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 1 --2- 172.21.15.29:0/1985103087 >> v2:172.21.15.29:3300/0 conn(0x7feda4149380 0x7feda41500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 1 -- 172.21.15.29:0/1985103087 >> v1:172.21.15.52:6789/0 conn(0x7feda40aa1a0 legacy=0x7feda4148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 1 monclient: found mon.noname-c 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.137+0000 7feda9852640 1 -- 172.21.15.29:0/1985103087 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda40a91f0 con 0x7feda40aafd0 2023-08-22T23:22:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.139+0000 7f0cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.139+0000 7f0cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.139+0000 7f0cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.140+0000 7feda9852640 1 --2- 172.21.15.29:0/1985103087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aafd0 0x7feda40ae380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feda40a9da0 tx=0x7feda002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b879280d528746d1 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.140+0000 7f0fe7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 0x7f0fe00c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.140+0000 7fe1c9a1e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c406b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c406b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43476/0 (socket says 172.21.15.29:43476) 2023-08-22T23:22:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/1394805763 learned_addr learned my addr 172.21.15.29:0/1394805763 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): get_auth_request con 0x7fe1c406f150 auth_method 0 2023-08-22T23:22:33.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7f0fe7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 0x7f0fe00c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43460/0 (socket says 172.21.15.29:43460) 2023-08-22T23:22:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7f0fe7fff640 1 -- 172.21.15.29:0/3254551592 learned_addr learned my addr 172.21.15.29:0/3254551592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7f0fe7fff640 10 monclient(hunting): get_auth_request con 0x7f0fe00392d0 auth_method 0 2023-08-22T23:22:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7f0fe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7f0fe7fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7f0fe7fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.141+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2023-08-22T23:22:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 1 -- 172.21.15.29:0/4237789787 >> v2:172.21.15.29:3300/0 conn(0x7f0cc8070350 msgr2=0x7f0cc813c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 1 --2- 172.21.15.29:0/4237789787 >> v2:172.21.15.29:3300/0 conn(0x7f0cc8070350 0x7f0cc813c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 1 -- 172.21.15.29:0/4237789787 >> v1:172.21.15.29:6789/0 conn(0x7f0cc813f460 legacy=0x7f0cc81424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 1 monclient: found mon.noname-c 2023-08-22T23:22:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0cbffff640 1 -- 172.21.15.29:0/4237789787 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc80aec50 con 0x7f0cc8070730 2023-08-22T23:22:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f976bfff640 1 --2- 172.21.15.29:0/503279459 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0ae730 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9764009e50 tx=0x7f976402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c079a84de44da4d2 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0fe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0fe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.143+0000 7f0fe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/1394805763 >> v2:172.21.15.29:3300/0 conn(0x7fe1c40acba0 msgr2=0x7fe1c4141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 1 --2- 172.21.15.29:0/1394805763 >> v2:172.21.15.29:3300/0 conn(0x7fe1c40acba0 0x7fe1c4141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/1394805763 >> v1:172.21.15.52:6789/0 conn(0x7fe1c40aeef0 legacy=0x7fe1c40af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 1 monclient: found mon.noname-b 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.145+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/1394805763 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c4070720 con 0x7fe1c406f150 2023-08-22T23:22:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.146+0000 7f93a6046640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.146+0000 7f93a6046640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.146+0000 7f93a6046640 10 monclient: monmap: 2023-08-22T23:22:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.148031+0000 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.148031+0000 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.175 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.147+0000 7f93a6046640 1 Processor -- start 2023-08-22T23:22:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.147+0000 7f93a6046640 1 -- start start 2023-08-22T23:22:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.147+0000 7f93a6046640 10 monclient: init 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 1 -- 172.21.15.29:0/3254551592 >> v2:172.21.15.29:3300/0 conn(0x7f0fe0038ef0 msgr2=0x7f0fe00c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 1 --2- 172.21.15.29:0/3254551592 >> v2:172.21.15.29:3300/0 conn(0x7f0fe0038ef0 0x7f0fe00c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 1 -- 172.21.15.29:0/3254551592 >> v1:172.21.15.29:6789/0 conn(0x7f0fe00c8310 legacy=0x7f0fe00cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 1 monclient: found mon.noname-b 2023-08-22T23:22:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f0fe7fff640 1 -- 172.21.15.29:0/3254551592 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0036590 con 0x7f0fe00392d0 2023-08-22T23:22:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 1 -- 172.21.15.29:0/1985103087 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00317e0 con 0x7feda40aafd0 2023-08-22T23:22:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: dump: 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 1 -- 172.21.15.29:0/1985103087 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda0031940 con 0x7feda40aafd0 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 1 -- 172.21.15.29:0/1985103087 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda0031cd0 con 0x7feda40aafd0 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7feda9051640 10 monclient: dump: 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.180 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.181 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7fedac2b7640 5 monclient: authenticate success, global_id 4318 2023-08-22T23:22:33.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7fedac2b7640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7fedac2b7640 4 set_mon_vals no callback set 2023-08-22T23:22:33.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.148+0000 7f93a6046640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-08-22T23:22:33.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f93a01448e0 0x7f93a0144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): picked mon.noname-a con 0x7f93a01448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a0145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): picked mon.noname-b con 0x7f93a00add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f93a013e190 0x7f93a013e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): picked mon.noname-c con 0x7f93a013e190 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a013eb50 con 0x7f93a01448e0 2023-08-22T23:22:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a00b12d0 con 0x7f93a013e190 2023-08-22T23:22:33.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00afae0 con 0x7f93a00add90 2023-08-22T23:22:33.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7f93a6046640 10 monclient(hunting): authenticate will time out at 1517.400087s 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7fedac2b7640 10 monclient: shutdown 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7fedac2b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7fedac2b7640 1 -- 172.21.15.29:0/1985103087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aafd0 msgr2=0x7feda40ae380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.149+0000 7fedac2b7640 1 --2- 172.21.15.29:0/1985103087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aafd0 0x7feda40ae380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feda40a9da0 tx=0x7feda002b880 comp rx=0 tx=0).stop 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 1 -- 172.21.15.29:0/1985103087 shutdown_connections 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 1 --2- 172.21.15.29:0/1985103087 >> v2:172.21.15.29:3300/0 conn(0x7feda4149380 0x7feda41500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 1 --2- 172.21.15.29:0/1985103087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aafd0 0x7feda40ae380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 1 -- 172.21.15.29:0/1985103087 >> 172.21.15.29:0/1985103087 conn(0x7feda40b27e0 msgr2=0x7feda40b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 1 -- 172.21.15.29:0/1985103087 shutdown_connections 2023-08-22T23:22:33.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 1 -- 172.21.15.29:0/1985103087 wait complete. 2023-08-22T23:22:33.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f93a013e190 0x7f93a013e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.150+0000 7fedac2b7640 10 monclient: monmap: 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 1 librados: starting msgr at 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 1 librados: starting objecter 2023-08-22T23:22:33.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 1 Processor -- start 2023-08-22T23:22:33.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 1 -- start start 2023-08-22T23:22:33.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 1 librados: setting wanted keys 2023-08-22T23:22:33.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 1 librados: calling monclient init 2023-08-22T23:22:33.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 10 monclient: init 2023-08-22T23:22:33.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f0cbffff640 1 --2- 172.21.15.29:0/4237789787 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8070730 0x7f0cc813ef20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80b5680 tx=0x7f0cb802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b48cad582d62f71 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f93a013e190 0x7f93a013e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39656/0 (socket says 172.21.15.29:39656) 2023-08-22T23:22:33.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 1 -- 172.21.15.29:0/2475181746 learned_addr learned my addr 172.21.15.29:0/2475181746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 10 monclient(hunting): get_auth_request con 0x7f93a013e190 auth_method 0 2023-08-22T23:22:33.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7f939ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.151+0000 7fedac2b7640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:33.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 0x7feda406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): picked mon.noname-b con 0x7feda40aa1a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40aafd0 0x7feda406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): picked mon.noname-a con 0x7feda40aafd0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feda4149380 0x7feda41de1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): picked mon.noname-c con 0x7feda4149380 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda40a5040 con 0x7feda40aafd0 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda41dddb0 con 0x7feda40aa1a0 2023-08-22T23:22:33.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda406ec30 con 0x7feda4149380 2023-08-22T23:22:33.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.152+0000 7fedac2b7640 10 monclient(hunting): authenticate will time out at 1517.402565s 2023-08-22T23:22:33.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.153+0000 7f939ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.153+0000 7f939ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.153+0000 7f939ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.153+0000 7fb70895d640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.153+0000 7fb70895d640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.153+0000 7fb70895d640 10 monclient: monmap: 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.154954+0000 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.154954+0000 2023-08-22T23:22:33.196 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.154+0000 7fb70895d640 1 Processor -- start 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.154+0000 7fb70895d640 1 -- start start 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.154+0000 7fb70895d640 10 monclient: init 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7041448e0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb7040add90 0x7fb70413e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7040add90 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:33.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb70414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): picked mon.noname-b con 0x7fb70413e6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7040b12d0 con 0x7fb7041448e0 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7040afae0 con 0x7fb7040add90 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040afdc0 con 0x7fb70413e6d0 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fb70895d640 10 monclient(hunting): authenticate will time out at 1517.405862s 2023-08-22T23:22:33.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 0x7feda406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 0x7feda406d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39658/0 (socket says 172.21.15.29:39658) 2023-08-22T23:22:33.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 1 -- 172.21.15.29:0/186695482 learned_addr learned my addr 172.21.15.29:0/186695482 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 10 monclient(hunting): get_auth_request con 0x7feda40aa1a0 auth_method 0 2023-08-22T23:22:33.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.155+0000 7fedaa053640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.156+0000 7f0fe7fff640 1 --2- 172.21.15.29:0/3254551592 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 0x7f0fe00c7dd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009870 tx=0x7f0fd402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0774053c866fa9b server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.157+0000 7fe1c9a1e640 1 --2- 172.21.15.29:0/1394805763 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c406b130 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009870 tx=0x7fe1c002d9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fbd7cfd23bc3bfd server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2023-08-22T23:22:33.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 1 -- 172.21.15.29:0/503279459 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976402ca70 con 0x7f976c0adfc0 2023-08-22T23:22:33.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 1 -- 172.21.15.29:0/2475181746 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 msgr2=0x7f93a0145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 1 --2- 172.21.15.29:0/2475181746 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a0145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:33.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: dump: 2023-08-22T23:22:33.203 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.204 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 1 -- 172.21.15.29:0/503279459 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976402cbd0 con 0x7f976c0adfc0 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 1 -- 172.21.15.29:0/503279459 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640343f0 con 0x7f976c0adfc0 2023-08-22T23:22:33.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f976affd640 10 monclient: dump: 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.207 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.207 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.207 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.207 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 1 -- 172.21.15.29:0/2475181746 >> v2:172.21.15.29:3300/0 conn(0x7f93a01448e0 msgr2=0x7f93a0144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 1 --2- 172.21.15.29:0/2475181746 >> v2:172.21.15.29:3300/0 conn(0x7f93a01448e0 0x7f93a0144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 1 monclient: found mon.noname-c 2023-08-22T23:22:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f939ffff640 1 -- 172.21.15.29:0/2475181746 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a00afdc0 con 0x7f93a013e190 2023-08-22T23:22:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f9772807640 5 monclient: authenticate success, global_id 4271 2023-08-22T23:22:33.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f9772807640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f9772807640 4 set_mon_vals no callback set 2023-08-22T23:22:33.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 1 -- 172.21.15.29:0/4237789787 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031870 con 0x7f0cc8070730 2023-08-22T23:22:33.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: dump: 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.210 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 1 -- 172.21.15.29:0/4237789787 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb80319d0 con 0x7f0cc8070730 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 1 -- 172.21.15.29:0/4237789787 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031d60 con 0x7f0cc8070730 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0cbf7fe640 10 monclient: dump: 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.212 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0ccdf29640 5 monclient: authenticate success, global_id 4321 2023-08-22T23:22:33.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0ccdf29640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.158+0000 7f0ccdf29640 4 set_mon_vals no callback set 2023-08-22T23:22:33.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 10 monclient: shutdown 2023-08-22T23:22:33.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 -- 172.21.15.29:0/503279459 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 msgr2=0x7f976c0ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 --2- 172.21.15.29:0/503279459 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0ae730 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9764009e50 tx=0x7f976402bed0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7fedaa053640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7fedaa053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7fedaa053640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 -- 172.21.15.29:0/503279459 shutdown_connections 2023-08-22T23:22:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 --2- 172.21.15.29:0/503279459 >> v2:172.21.15.52:3300/0 conn(0x7f976c0713c0 0x7f976c0a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 --2- 172.21.15.29:0/503279459 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0ae730 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 -- 172.21.15.29:0/503279459 >> 172.21.15.29:0/503279459 conn(0x7f976c0aee40 msgr2=0x7f976c0af220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 -- 172.21.15.29:0/503279459 shutdown_connections 2023-08-22T23:22:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f9772807640 1 -- 172.21.15.29:0/503279459 wait complete. 2023-08-22T23:22:33.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 10 monclient: shutdown 2023-08-22T23:22:33.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 1 -- 172.21.15.29:0/4237789787 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8070730 msgr2=0x7f0cc813ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/4237789787 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8070730 0x7f0cc813ef20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80b5680 tx=0x7f0cb802b910 comp rx=0 tx=0).stop 2023-08-22T23:22:33.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 1 -- 172.21.15.29:0/4237789787 shutdown_connections 2023-08-22T23:22:33.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/4237789787 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8070730 0x7f0cc813ef20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/4237789787 >> v2:172.21.15.29:3300/0 conn(0x7f0cc8070350 0x7f0cc813c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.159+0000 7f0ccdf29640 1 -- 172.21.15.29:0/4237789787 >> 172.21.15.29:0/4237789787 conn(0x7f0cc80b4640 msgr2=0x7f0cc80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f0ccdf29640 1 -- 172.21.15.29:0/4237789787 shutdown_connections 2023-08-22T23:22:33.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f9772807640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f9772807640 10 monclient: monmap: 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.219 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.220 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.220 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f0ccdf29640 1 -- 172.21.15.29:0/4237789787 wait complete. 2023-08-22T23:22:33.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f9772807640 1 librados: starting msgr at 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f9772807640 1 librados: starting objecter 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f9772807640 1 Processor -- start 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f0ccdf29640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.160+0000 7f0ccdf29640 10 monclient: monmap: 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.221 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.222 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 -- start start 2023-08-22T23:22:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 librados: setting wanted keys 2023-08-22T23:22:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 librados: calling monclient init 2023-08-22T23:22:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient: init 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 1 librados: starting msgr at 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 1 librados: starting objecter 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 1 Processor -- start 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 1 -- start start 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 1 librados: setting wanted keys 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 1 librados: calling monclient init 2023-08-22T23:22:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f0ccdf29640 10 monclient: init 2023-08-22T23:22:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 0x7f976c147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): picked mon.noname-b con 0x7f976c0713c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f976c0a8a10 0x7f976c0a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): picked mon.noname-a con 0x7f976c0a8a10 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): picked mon.noname-c con 0x7f976c0adfc0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0ab060 con 0x7f976c0a8a10 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0b1550 con 0x7f976c0713c0 2023-08-22T23:22:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0b16c0 con 0x7f976c0adfc0 2023-08-22T23:22:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.161+0000 7f9772807640 10 monclient(hunting): authenticate will time out at 1517.412267s 2023-08-22T23:22:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0cc8070350 0x7f0cc8140e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc8070350 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc813f460 0x7f0cc81da7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc813f460 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 0x7f0cc80aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc8142af0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80b4bd0 con 0x7f0cc813f460 2023-08-22T23:22:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8140680 con 0x7f0cc8142af0 2023-08-22T23:22:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8140960 con 0x7f0cc8070350 2023-08-22T23:22:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.162+0000 7f0ccdf29640 10 monclient(hunting): authenticate will time out at 1517.413176s 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 1 -- 172.21.15.29:0/186695482 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feda4149380 msgr2=0x7feda41de1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feda4149380 0x7feda41de1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 1 -- 172.21.15.29:0/186695482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40aafd0 msgr2=0x7feda406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40aafd0 0x7feda406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 1 monclient: found mon.noname-b 2023-08-22T23:22:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.163+0000 7fedaa053640 1 -- 172.21.15.29:0/186695482 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00094d0 con 0x7feda40aa1a0 2023-08-22T23:22:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.166+0000 7f976bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 0x7f976c147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.167+0000 7f0ccd728640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 0x7f0cc80aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.168+0000 7f0ccd728640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 0x7f0cc80aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39668/0 (socket says 172.21.15.29:39668) 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.168+0000 7f0ccd728640 1 -- 172.21.15.29:0/2051262038 learned_addr learned my addr 172.21.15.29:0/2051262038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.168+0000 7f0ccd728640 10 monclient(hunting): get_auth_request con 0x7f0cc8142af0 auth_method 0 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.168+0000 7f0ccd728640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.168+0000 7f0ccd728640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.168+0000 7f0ccd728640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7f976bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 0x7f976c147d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39664/0 (socket says 172.21.15.29:39664) 2023-08-22T23:22:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7f976bfff640 1 -- 172.21.15.29:0/2766711473 learned_addr learned my addr 172.21.15.29:0/2766711473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7f976bfff640 10 monclient(hunting): get_auth_request con 0x7f976c0713c0 auth_method 0 2023-08-22T23:22:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7f976bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7f976bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7f976bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.169+0000 7fb6fbfff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb7040add90 0x7fb70413e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7fb702d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb70414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7f0ccd728640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7f0ccd728640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.170+0000 7f0ccd728640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 1 -- 172.21.15.29:0/2766711473 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 msgr2=0x7f976c0a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 1 -- 172.21.15.29:0/2766711473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f976c0a8a10 msgr2=0x7f976c0a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f976c0a8a10 0x7f976c0a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 1 monclient: found mon.noname-b 2023-08-22T23:22:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.173+0000 7f976bfff640 1 -- 172.21.15.29:0/2766711473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9764009b30 con 0x7f976c0713c0 2023-08-22T23:22:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7f0cccf27640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0cc8070350 0x7f0cc8140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7fb6fbfff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb7040add90 0x7fb70413e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39662/0 (socket says 172.21.15.29:39662) 2023-08-22T23:22:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7fb6fbfff640 1 -- 172.21.15.29:0/947497573 learned_addr learned my addr 172.21.15.29:0/947497573 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7fb6fbfff640 10 monclient(hunting): get_auth_request con 0x7fb7040add90 auth_method 0 2023-08-22T23:22:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7fb6fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7fb6fbfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.174+0000 7fb6fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.175+0000 7fedaa053640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 0x7feda406d940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00c800 tx=0x7fed9c00ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.175+0000 7fed9affd640 10 client.?.objecter ms_handle_connect 0x7feda40aa1a0 2023-08-22T23:22:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.175+0000 7fed9affd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb702d9d640 10 monclient(hunting): get_auth_request con 0x7fb70413e6d0 auth_method 0 2023-08-22T23:22:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb702d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb702d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb702d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7f0cccf27640 10 monclient(hunting): get_auth_request con 0x7f0cc8070350 auth_method 0 2023-08-22T23:22:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7f0cccf27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7f0cccf27640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7f0cccf27640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.176+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 1 -- 172.21.15.29:0/3254551592 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4031640 con 0x7f0fe00392d0 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: dump: 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 1 -- 172.21.15.29:0/3254551592 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd40317a0 con 0x7f0fe00392d0 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 1 -- 172.21.15.29:0/3254551592 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4031b30 con 0x7f0fe00392d0 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0feed19640 5 monclient: authenticate success, global_id 4277 2023-08-22T23:22:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0feed19640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0feed19640 4 set_mon_vals no callback set 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f0fe77fe640 10 monclient: dump: 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f976bfff640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 0x7f976c147d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9764036040 tx=0x7f9764004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f9768ff9640 10 client.?.objecter ms_handle_connect 0x7f976c0713c0 2023-08-22T23:22:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7f9768ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7fb702d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7fb702d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.178+0000 7fb702d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 10 monclient: shutdown 2023-08-22T23:22:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 -- 172.21.15.29:0/3254551592 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 msgr2=0x7f0fe00c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 --2- 172.21.15.29:0/3254551592 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 0x7f0fe00c7dd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009870 tx=0x7f0fd402b8f0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 -- 172.21.15.29:0/3254551592 shutdown_connections 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 --2- 172.21.15.29:0/3254551592 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe00392d0 0x7f0fe00c7dd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 --2- 172.21.15.29:0/3254551592 >> v2:172.21.15.29:3300/0 conn(0x7f0fe0038ef0 0x7f0fe00c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 -- 172.21.15.29:0/3254551592 >> 172.21.15.29:0/3254551592 conn(0x7f0fe0006180 msgr2=0x7f0fe0006590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 -- 172.21.15.29:0/3254551592 shutdown_connections 2023-08-22T23:22:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.179+0000 7f0feed19640 1 -- 172.21.15.29:0/3254551592 wait complete. 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 1 -- 172.21.15.29:0/947497573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 msgr2=0x7fb70414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 1 --2- 172.21.15.29:0/947497573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb70414ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 1 -- 172.21.15.29:0/947497573 >> v1:172.21.15.29:6789/0 conn(0x7fb7041448e0 legacy=0x7fb704144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 1 monclient: found mon.noname-c 2023-08-22T23:22:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7fb6fbfff640 1 -- 172.21.15.29:0/947497573 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7040b00a0 con 0x7fb7040add90 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7f0feed19640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7f0feed19640 10 monclient: monmap: 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7f0feed19640 1 librados: starting msgr at 2023-08-22T23:22:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7f0feed19640 1 librados: starting objecter 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.180+0000 7f0feed19640 1 Processor -- start 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.181+0000 7f0feed19640 1 -- start start 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.181+0000 7f0feed19640 1 librados: setting wanted keys 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.181+0000 7f0feed19640 1 librados: calling monclient init 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.181+0000 7f0feed19640 10 monclient: init 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:22:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 0x7f0fe00cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): picked mon.noname-c con 0x7f0fe0038ef0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe00c8310 0x7f0fe0009d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe00c8310 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe0030500 0x7f0fe016cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe0030500 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0030070 con 0x7f0fe0030500 2023-08-22T23:22:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe002ed30 con 0x7f0fe00c8310 2023-08-22T23:22:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe000de20 con 0x7f0fe0038ef0 2023-08-22T23:22:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0feed19640 10 monclient(hunting): authenticate will time out at 1517.432760s 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0cccf27640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0cccf27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.182+0000 7f0cccf27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0cbffff640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc813f460 0x7f0cc81da7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 1 -- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0cc8070350 msgr2=0x7f0cc8140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0cc8070350 0x7f0cc8140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 1 -- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc813f460 msgr2=0x7f0cc81da7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc813f460 0x7f0cc81da7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 1 monclient: found mon.noname-b 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7f0ccd728640 1 -- 172.21.15.29:0/2051262038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb80094d0 con 0x7f0cc8142af0 2023-08-22T23:22:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 1 -- 172.21.15.29:0/1394805763 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0031810 con 0x7fe1c406f150 2023-08-22T23:22:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: dump: 2023-08-22T23:22:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 1 -- 172.21.15.29:0/1394805763 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c0031970 con 0x7fe1c406f150 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 1 -- 172.21.15.29:0/1394805763 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0031d00 con 0x7fe1c406f150 2023-08-22T23:22:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.184+0000 7fe1c921d640 10 monclient: dump: 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 5 monclient: authenticate success, global_id 4274 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 4 set_mon_vals no callback set 2023-08-22T23:22:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 0x7f0fe00cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 10 monclient: shutdown 2023-08-22T23:22:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 1 -- 172.21.15.29:0/1394805763 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 msgr2=0x7fe1c406b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7fe1cc483640 1 --2- 172.21.15.29:0/1394805763 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c406b130 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009870 tx=0x7fe1c002d9f0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 0x7f0fe00cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43530/0 (socket says 172.21.15.29:43530) 2023-08-22T23:22:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 1 -- 172.21.15.29:0/1588893936 learned_addr learned my addr 172.21.15.29:0/1588893936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 10 monclient(hunting): get_auth_request con 0x7f0fe0038ef0 auth_method 0 2023-08-22T23:22:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.185+0000 7f0fecab5640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7fe1cc483640 1 -- 172.21.15.29:0/1394805763 shutdown_connections 2023-08-22T23:22:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7fe1cc483640 1 --2- 172.21.15.29:0/1394805763 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c406b130 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7fe1cc483640 1 --2- 172.21.15.29:0/1394805763 >> v2:172.21.15.29:3300/0 conn(0x7fe1c40acba0 0x7fe1c4141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7fe1cc483640 1 -- 172.21.15.29:0/1394805763 >> 172.21.15.29:0/1394805763 conn(0x7fe1c40b0860 msgr2=0x7fe1c406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7fe1cc483640 1 -- 172.21.15.29:0/1394805763 shutdown_connections 2023-08-22T23:22:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7fe1cc483640 1 -- 172.21.15.29:0/1394805763 wait complete. 2023-08-22T23:22:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7f0fecab5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7f0fecab5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.186+0000 7f0fecab5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 10 monclient: monmap: 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 1 librados: starting msgr at 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 1 librados: starting objecter 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 1 Processor -- start 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 1 -- start start 2023-08-22T23:22:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 1 librados: setting wanted keys 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 1 librados: calling monclient init 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.187+0000 7fe1cc483640 10 monclient: init 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c413d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): picked mon.noname-c con 0x7fe1c406f150 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c40acba0 0x7fe1c41376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): picked mon.noname-a con 0x7fe1c40acba0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1c40aeef0 0x7fe1c4137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): picked mon.noname-b con 0x7fe1c40aeef0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c4138130 con 0x7fe1c40acba0 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c413eaa0 con 0x7fe1c40aeef0 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c413cec0 con 0x7fe1c406f150 2023-08-22T23:22:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.188+0000 7fe1cc483640 10 monclient(hunting): authenticate will time out at 1517.438620s 2023-08-22T23:22:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.189+0000 7fb6fbfff640 1 --2- 172.21.15.29:0/947497573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 0x7fb70413e190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009870 tx=0x7fb6ec02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86187efdcc12c2de server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.190+0000 7f0ccd728640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 0x7f0cc80aa510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000b990 tx=0x7f0cc000bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.190+0000 7f0cbd7fa640 10 client.?.objecter ms_handle_connect 0x7f0cc8142af0 2023-08-22T23:22:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.190+0000 7f0cbd7fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976403e450 con 0x7f976c0713c0 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 10 monclient: dump: 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.276 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9768ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 5 monclient: authenticate success, global_id 4330 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 10 monclient: _renew_subs 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976c0a9020 con 0x7f976c0713c0 2023-08-22T23:22:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 10 monclient: _renew_subs 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976c0ab580 con 0x7f976c0713c0 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.191+0000 7f9772807640 1 librados: init done 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.192+0000 7f9772807640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.192+0000 7f9772807640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.192+0000 7f9772807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.192+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976c074db0 con 0x7f976c0713c0 2023-08-22T23:22:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-08-22T23:22:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 1 -- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe00c8310 msgr2=0x7f0fe0009d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe00c8310 0x7f0fe0009d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 1 -- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe0030500 msgr2=0x7f0fe016cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe0030500 0x7f0fe016cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 1 monclient: found mon.noname-c 2023-08-22T23:22:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f0fecab5640 1 -- 172.21.15.29:0/1588893936 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40094d0 con 0x7f0fe0038ef0 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c003d80 con 0x7feda40aa1a0 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: dump: 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c005070 con 0x7feda40aa1a0 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c005420 con 0x7feda40aa1a0 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fed9affd640 10 monclient: dump: 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.285 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7f07fd6b5640 10 monclient: get_monmap_and_config 2023-08-22T23:22:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.194+0000 7f07fd6b5640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.194+0000 7f07fd6b5640 10 monclient: monmap: 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:33.195325+0000 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:33.195325+0000 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:33.287 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 5 monclient: authenticate success, global_id 4327 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 10 monclient: _renew_subs 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda41de710 con 0x7feda40aa1a0 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7feda8850640 4 set_mon_vals no callback set 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 10 monclient: _renew_subs 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda41dda60 con 0x7feda40aa1a0 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.193+0000 7fedac2b7640 1 librados: init done 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7fedac2b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7fedac2b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7fedac2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda41ddef0 con 0x7feda40aa1a0 2023-08-22T23:22:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 Processor -- start 2023-08-22T23:22:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 -- start start 2023-08-22T23:22:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient: init 2023-08-22T23:22:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07f81448e0 0x7f07f8144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): picked mon.noname-a con 0x7f07f81448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07f80add90 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f07f813e8a0 0x7f07f8145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): picked mon.noname-c con 0x7f07f813e8a0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f80b12d0 con 0x7f07f80add90 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f80afae0 con 0x7f07f81448e0 2023-08-22T23:22:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f80afdc0 con 0x7f07f813e8a0 2023-08-22T23:22:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.195+0000 7f07fd6b5640 10 monclient(hunting): authenticate will time out at 1517.446312s 2023-08-22T23:22:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.200+0000 7f0fecab5640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 0x7f0fe00cbd80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b4a0 tx=0x7f0fdc00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.200+0000 7f0fe57fa640 10 client.?.objecter ms_handle_connect 0x7f0fe0038ef0 2023-08-22T23:22:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.200+0000 7f0fe57fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.204+0000 7fe1ca21f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c413d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc010610 con 0x7f0fe0038ef0 2023-08-22T23:22:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: dump: 2023-08-22T23:22:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.295 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc010770 con 0x7f0fe0038ef0 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc010b20 con 0x7f0fe0038ef0 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe57fa640 10 monclient: dump: 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 5 monclient: authenticate success, global_id 4286 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 10 monclient: _renew_subs 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe016d1e0 con 0x7f0fe0038ef0 2023-08-22T23:22:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0fe6ffd640 4 set_mon_vals no callback set 2023-08-22T23:22:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 10 monclient: _renew_subs 2023-08-22T23:22:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe002d6b0 con 0x7f0fe0038ef0 2023-08-22T23:22:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7f0feed19640 1 librados: init done 2023-08-22T23:22:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7fe1ca21f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c413d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43532/0 (socket says 172.21.15.29:43532) 2023-08-22T23:22:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7fe1ca21f640 1 -- 172.21.15.29:0/2478429672 learned_addr learned my addr 172.21.15.29:0/2478429672 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7fe1ca21f640 10 monclient(hunting): get_auth_request con 0x7fe1c406f150 auth_method 0 2023-08-22T23:22:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7fe1ca21f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7fe1ca21f640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.205+0000 7fe1ca21f640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.206+0000 7f0feed19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.206+0000 7f0feed19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.206+0000 7f0feed19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.206+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe00c5860 con 0x7f0fe0038ef0 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.207+0000 7fe1ca21f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.207+0000 7fe1ca21f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.207+0000 7fe1ca21f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.270+0000 7fe1c9a1e640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c40acba0 0x7fe1c41376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.271+0000 7fe1c9a1e640 10 monclient(hunting): get_auth_request con 0x7fe1c40acba0 auth_method 0 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.271+0000 7fe1c9a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.271+0000 7fe1c9a1e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.271+0000 7fe1c9a1e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.272+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.272+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.272+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.272+0000 7f07effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f07f80add90 0x7f07f813e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59984/0 (socket says 172.21.15.29:59984) 2023-08-22T23:22:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.272+0000 7f07effff640 1 -- 172.21.15.29:0/4189488221 learned_addr learned my addr 172.21.15.29:0/4189488221 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2023-08-22T23:22:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 msgr2=0x7fe1c413d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c413d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1c40aeef0 msgr2=0x7fe1c4137bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1c40aeef0 0x7fe1c4137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 1 monclient: found mon.noname-a 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.275+0000 7fe1c9a1e640 1 -- 172.21.15.29:0/2478429672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c00094d0 con 0x7fe1c40acba0 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.279+0000 7fe1c9a1e640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c40acba0 0x7fe1c41376b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40706f0 tx=0x7fe1c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.279+0000 7fe1baffd640 10 client.?.objecter ms_handle_connect 0x7fe1c40acba0 2023-08-22T23:22:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.279+0000 7fe1baffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.284+0000 7f07f6ffd640 1 --2- 172.21.15.29:0/4189488221 >> v2:172.21.15.29:3300/0 conn(0x7f07f81448e0 0x7f07f8144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f67fc640 1 -- 172.21.15.29:0/4189488221 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221103148 0 0) 0x7f07f80b12d0 con 0x7f07f80add90 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f67fc640 10 monclient(hunting): _negotiate 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f67fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f67fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f67fc640 10 monclient(hunting): my global_id is 4320 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f67fc640 1 -- 172.21.15.29:0/4189488221 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e4003520 con 0x7f07f80add90 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f6ffd640 10 monclient(hunting): get_auth_request con 0x7f07f81448e0 auth_method 0 2023-08-22T23:22:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f6ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.286+0000 7f07f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.289+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.289+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.289+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2023-08-22T23:22:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 1 -- 172.21.15.29:0/4189488221 >> v2:172.21.15.52:3300/0 conn(0x7f07f813e8a0 msgr2=0x7f07f8145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 1 --2- 172.21.15.29:0/4189488221 >> v2:172.21.15.52:3300/0 conn(0x7f07f813e8a0 0x7f07f8145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 1 -- 172.21.15.29:0/4189488221 >> v1:172.21.15.29:6789/0 conn(0x7f07f80add90 legacy=0x7f07f813e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 1 monclient: found mon.noname-a 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.290+0000 7f07f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.291+0000 7f07f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.291+0000 7f07f6ffd640 1 -- 172.21.15.29:0/4189488221 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80b00a0 con 0x7f07f81448e0 2023-08-22T23:22:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c003d070 con 0x7fe1c40acba0 2023-08-22T23:22:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: dump: 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c00394c0 con 0x7fe1c40acba0 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0039870 con 0x7fe1c40acba0 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1baffd640 10 monclient: dump: 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 5 monclient: authenticate success, global_id 4317 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 10 monclient: _renew_subs 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1c8a1c640 4 set_mon_vals no callback set 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c413bf00 con 0x7fe1c40acba0 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 10 monclient: _renew_subs 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c413de70 con 0x7fe1c40acba0 2023-08-22T23:22:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.295+0000 7fe1cc483640 1 librados: init done 2023-08-22T23:22:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.296+0000 7fe1cc483640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.296+0000 7fe1cc483640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.296+0000 7fe1cc483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.296+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c41d1840 con 0x7fe1c40acba0 2023-08-22T23:22:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.299+0000 7f07f6ffd640 1 --2- 172.21.15.29:0/4189488221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f81448e0 0x7f07f8144cc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07f80b07a0 tx=0x7f07e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496d34559b5a3321 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.442+0000 7f939ffff640 1 --2- 172.21.15.29:0/2475181746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 0x7f93a013e610 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f93a00ad970 tx=0x7f939802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b231dff908230e8 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976403e5b0 con 0x7f976c0713c0 2023-08-22T23:22:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f9768ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976403e920 con 0x7f976c0713c0 2023-08-22T23:22:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f9768ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f9768ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f9768ff9640 10 monclient: dump: 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.460 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.456+0000 7f976a7fc640 4 set_mon_vals no callback set 2023-08-22T23:22:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.457+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f976403ea80 con 0x7f976c0713c0 2023-08-22T23:22:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.458+0000 7f9768ff9640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f975806b9e0 0x7f975806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.458+0000 7f976b7fe640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f975806b9e0 0x7f975806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.458+0000 7f976b7fe640 10 monclient: get_auth_request con 0x7f975806b9e0 auth_method 0 2023-08-22T23:22:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f976b7fe640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f975806b9e0 0x7f975806dea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9760006fd0 tx=0x7f9760008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f9768ff9640 10 client.4330.objecter ms_handle_connect 0x7f975806b9e0 2023-08-22T23:22:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f976407b4e0 con 0x7f976c0713c0 2023-08-22T23:22:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f9768ff9640 10 client.4330.objecter ms_dispatch 0x7f976c000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f9768ff9640 3 client.4330.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f9768ff9640 3 client.4330.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.459+0000 7f9768ff9640 20 client.4330.objecter dump_active .. 0 homeless 2023-08-22T23:22:33.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.470+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f9764077120 con 0x7f976c0713c0 2023-08-22T23:22:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.470+0000 7f9768ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.470+0000 7f9768ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:33.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.498+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0fdc010c80 con 0x7f0fe0038ef0 2023-08-22T23:22:33.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.498+0000 7f0fe57fa640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0fcc06b9e0 0x7f0fcc06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.498+0000 7f0fe7fff640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0fcc06b9e0 0x7f0fcc06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.498+0000 7f0fe7fff640 10 monclient: get_auth_request con 0x7f0fcc06b9e0 auth_method 0 2023-08-22T23:22:33.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.499+0000 7f0fe7fff640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0fcc06b9e0 0x7f0fcc06dea0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036000 tx=0x7f0fd4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.499+0000 7f0fe57fa640 10 client.4286.objecter ms_handle_connect 0x7f0fcc06b9e0 2023-08-22T23:22:33.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0fdc0580a0 con 0x7f0fe0038ef0 2023-08-22T23:22:33.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 10 client.4286.objecter ms_dispatch 0x7f0fe8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:33.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 3 client.4286.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:33.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 3 client.4286.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:33.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 20 client.4286.objecter dump_active .. 0 homeless 2023-08-22T23:22:33.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f0fdc053d50 con 0x7f0fe0038ef0 2023-08-22T23:22:33.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.515+0000 7f0fe57fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.591+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe1c00399d0 con 0x7fe1c40acba0 2023-08-22T23:22:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7fe1baffd640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe1a806ba30 0x7fe1a806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 1 -- 172.21.15.29:0/4189488221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e802e070 con 0x7f07f81448e0 2023-08-22T23:22:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: dump: 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.596 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 1 -- 172.21.15.29:0/4189488221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e80315a0 con 0x7f07f81448e0 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 5 monclient: authenticate success, global_id 4323 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 1 -- 172.21.15.29:0/4189488221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8031930 con 0x7f07f81448e0 2023-08-22T23:22:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 4 set_mon_vals no callback set 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07f67fc640 10 monclient: dump: 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.599 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7fe1ca21f640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe1a806ba30 0x7fe1a806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7fe1ca21f640 10 monclient: get_auth_request con 0x7fe1a806ba30 auth_method 0 2023-08-22T23:22:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7fe1baffd640 10 client.4317.objecter ms_handle_connect 0x7fe1a806ba30 2023-08-22T23:22:33.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7fe1ca21f640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe1a806ba30 0x7fe1a806def0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009ac0 tx=0x7fe1b40091c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 10 monclient: shutdown 2023-08-22T23:22:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 1 -- 172.21.15.29:0/4189488221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f81448e0 msgr2=0x7f07f8144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.592+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/4189488221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f81448e0 0x7f07f8144cc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07f80b07a0 tx=0x7f07e802b840 comp rx=0 tx=0).stop 2023-08-22T23:22:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 1 -- 172.21.15.29:0/4189488221 shutdown_connections 2023-08-22T23:22:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/4189488221 >> v2:172.21.15.52:3300/0 conn(0x7f07f813e8a0 0x7f07f8145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/4189488221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f81448e0 0x7f07f8144cc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 1 -- 172.21.15.29:0/4189488221 >> 172.21.15.29:0/4189488221 conn(0x7f07f8065ed0 msgr2=0x7f07f80b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 1 -- 172.21.15.29:0/4189488221 shutdown_connections 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 1 -- 172.21.15.29:0/4189488221 wait complete. 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.593+0000 7f07fd6b5640 10 monclient: monmap: 2023-08-22T23:22:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 librados: starting msgr at 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 librados: starting objecter 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 Processor -- start 2023-08-22T23:22:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 -- start start 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 librados: setting wanted keys 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 librados: calling monclient init 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient: init 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 0x7f07f8135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): picked mon.noname-c con 0x7f07f80add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f813e8a0 0x7f07f8136720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): picked mon.noname-a con 0x7f07f813e8a0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07f81448e0 0x7f07f8132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): picked mon.noname-b con 0x7f07f81448e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f81396d0 con 0x7f07f813e8a0 2023-08-22T23:22:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8134d40 con 0x7f07f81448e0 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8134eb0 con 0x7f07f80add90 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.594+0000 7f07fd6b5640 10 monclient(hunting): authenticate will time out at 1517.845299s 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 0x7f07f8135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 0x7f07f8135e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43540/0 (socket says 172.21.15.29:43540) 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 1 -- 172.21.15.29:0/2182184821 learned_addr learned my addr 172.21.15.29:0/2182184821 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 10 monclient(hunting): get_auth_request con 0x7f07f80add90 auth_method 0 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.595+0000 7f07f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.596+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.596+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.596+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-08-22T23:22:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 1 -- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07f81448e0 msgr2=0x7f07f8132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07f81448e0 0x7f07f8132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 1 -- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f813e8a0 msgr2=0x7f07f8136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f813e8a0 0x7f07f8136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 1 monclient: found mon.noname-c 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.599+0000 7f07f6ffd640 1 -- 172.21.15.29:0/2182184821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8009400 con 0x7f07f80add90 2023-08-22T23:22:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.601+0000 7f07f6ffd640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 0x7f07f8135e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009890 tx=0x7f07e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.601+0000 7f07ef7fe640 10 client.?.objecter ms_handle_connect 0x7f07f80add90 2023-08-22T23:22:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.601+0000 7f07ef7fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8032840 con 0x7f07f80add90 2023-08-22T23:22:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: dump: 2023-08-22T23:22:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.617 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e80329a0 con 0x7f07f80add90 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 5 monclient: authenticate success, global_id 4292 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 10 monclient: _renew_subs 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f8136c60 con 0x7f07f80add90 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8032d50 con 0x7f07f80add90 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07ef7fe640 10 monclient: dump: 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 10 monclient: _renew_subs 2023-08-22T23:22:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f8133140 con 0x7f07f80add90 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07fd6b5640 1 librados: init done 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.605+0000 7f07f5ffb640 4 set_mon_vals no callback set 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.606+0000 7f07fd6b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.606+0000 7f07fd6b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.606+0000 7f07fd6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.606+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f813f640 con 0x7f07f80add90 2023-08-22T23:22:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe1c007ab30 con 0x7fe1c40acba0 2023-08-22T23:22:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 10 client.4317.objecter ms_dispatch 0x7fe1c4000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 3 client.4317.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 3 client.4317.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 20 client.4317.objecter dump_active .. 0 homeless 2023-08-22T23:22:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fe1c00767e0 con 0x7fe1c40acba0 2023-08-22T23:22:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.617+0000 7fe1baffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:33.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.632+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f07e804d3f0 con 0x7f07f80add90 2023-08-22T23:22:33.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.632+0000 7f07ef7fe640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f07d806b9e0 0x7f07d806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.632+0000 7f07effff640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f07d806b9e0 0x7f07d806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.633+0000 7f07effff640 10 monclient: get_auth_request con 0x7f07d806b9e0 auth_method 0 2023-08-22T23:22:33.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.633+0000 7f07ef7fe640 10 client.4292.objecter ms_handle_connect 0x7f07d806b9e0 2023-08-22T23:22:33.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.633+0000 7f07effff640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f07d806b9e0 0x7f07d806dea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f07f8137950 tx=0x7f07e0007930 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000caa0 con 0x7f0cc8142af0 2023-08-22T23:22:33.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 10 monclient: dump: 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.676 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0cbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 5 monclient: authenticate success, global_id 4333 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 10 monclient: _renew_subs 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc81402a0 con 0x7f0cc8142af0 2023-08-22T23:22:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 10 monclient: _renew_subs 2023-08-22T23:22:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc81daef0 con 0x7f0cc8142af0 2023-08-22T23:22:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.672+0000 7f0ccdf29640 1 librados: init done 2023-08-22T23:22:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.674+0000 7f0ccdf29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.674+0000 7f0ccdf29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.674+0000 7f0ccdf29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.674+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc8074db0 con 0x7f0cc8142af0 2023-08-22T23:22:33.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.691+0000 7f9772807640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-08-22T23:22:33.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.691+0000 7f9772807640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-08-22T23:22:33.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.691+0000 7f9772807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:33.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.691+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f976c0a5cb0 con 0x7f976c0713c0 2023-08-22T23:22:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc00098d0 con 0x7f0cc8142af0 2023-08-22T23:22:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbd7fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000ce00 con 0x7f0cc8142af0 2023-08-22T23:22:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbd7fa640 10 monclient: dump: 2023-08-22T23:22:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.715 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.711+0000 7f0cbeffd640 4 set_mon_vals no callback set 2023-08-22T23:22:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.754+0000 7f0feed19640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-08-22T23:22:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.754+0000 7f0feed19640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-08-22T23:22:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.754+0000 7f0feed19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.754+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0fe000dfb0 con 0x7f0fe0038ef0 2023-08-22T23:22:33.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.826+0000 7fe1cc483640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-08-22T23:22:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.826+0000 7fe1cc483640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-08-22T23:22:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.826+0000 7fe1cc483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.826+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe1c41d1840 con 0x7fe1c40acba0 2023-08-22T23:22:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.830+0000 7fe1baffd640 1 -- 172.21.15.29:0/2478429672 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1c41d1840 con 0x7fe1c40acba0 2023-08-22T23:22:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.830+0000 7fe1baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-08-22T23:22:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.830+0000 7fe1baffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:33.833 INFO:teuthology.orchestra.run.smithi029.stdout:64424509444 2023-08-22T23:22:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 10 librados: watch_flush enter 2023-08-22T23:22:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 10 librados: watch_flush exit 2023-08-22T23:22:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 20 client.4317.objecter shutdown clearing up homeless session... 2023-08-22T23:22:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 10 client.4317.objecter successfully canceled tick 2023-08-22T23:22:33.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe1a806ba30 msgr2=0x7fe1a806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe1a806ba30 0x7fe1a806def0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009ac0 tx=0x7fe1b40091c0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 10 monclient: shutdown 2023-08-22T23:22:33.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c40acba0 msgr2=0x7fe1c41376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c40acba0 0x7fe1c41376b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40706f0 tx=0x7fe1c00046c0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 shutdown_connections 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe1a806ba30 0x7fe1a806def0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1c40aeef0 0x7fe1c4137bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c40acba0 0x7fe1c41376b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 --2- 172.21.15.29:0/2478429672 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe1c406f150 0x7fe1c413d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.832+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 >> 172.21.15.29:0/2478429672 conn(0x7fe1c40b0860 msgr2=0x7fe1c413aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.833+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 shutdown_connections 2023-08-22T23:22:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.833+0000 7fe1cc483640 1 -- 172.21.15.29:0/2478429672 wait complete. 2023-08-22T23:22:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.833+0000 7fe1cc483640 1 librados: shutdown 2023-08-22T23:22:33.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.4 2023-08-22T23:22:33.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f07e8079dd0 con 0x7f07f80add90 2023-08-22T23:22:33.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 10 client.4292.objecter ms_dispatch 0x7f07f8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:33.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 3 client.4292.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:33.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 3 client.4292.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:33.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 20 client.4292.objecter dump_active .. 0 homeless 2023-08-22T23:22:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f07e8075a80 con 0x7f07f80add90 2023-08-22T23:22:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.915+0000 7f07ef7fe640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 1 -- 172.21.15.29:0/947497573 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec031640 con 0x7fb7040add90 2023-08-22T23:22:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 10 monclient: dump: 2023-08-22T23:22:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.924 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 10 monclient: _finish_auth 0 2023-08-22T23:22:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 10 monclient: _check_auth_tickets 2023-08-22T23:22:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb701d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb70895d640 5 monclient: authenticate success, global_id 4336 2023-08-22T23:22:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.920+0000 7fb70895d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.925+0000 7f0fe57fa640 1 -- 172.21.15.29:0/1588893936 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0fdc01e090 con 0x7f0fe0038ef0 2023-08-22T23:22:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.925+0000 7f0fe57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-08-22T23:22:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.925+0000 7f0fe57fa640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:33.927 INFO:teuthology.orchestra.run.smithi029.stdout:60129542148 2023-08-22T23:22:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 10 librados: watch_flush enter 2023-08-22T23:22:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 10 librados: watch_flush exit 2023-08-22T23:22:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 20 client.4286.objecter shutdown clearing up homeless session... 2023-08-22T23:22:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 10 client.4286.objecter successfully canceled tick 2023-08-22T23:22:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0fcc06b9e0 msgr2=0x7f0fcc06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0fcc06b9e0 0x7f0fcc06dea0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036000 tx=0x7f0fd4030000 comp rx=0 tx=0).stop 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 10 monclient: shutdown 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 msgr2=0x7f0fe00cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 0x7f0fe00cbd80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b4a0 tx=0x7f0fdc00b960 comp rx=0 tx=0).stop 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 shutdown_connections 2023-08-22T23:22:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0fcc06b9e0 0x7f0fcc06dea0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe0030500 0x7f0fe016cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe00c8310 0x7f0fe0009d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 --2- 172.21.15.29:0/1588893936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0fe0038ef0 0x7f0fe00cbd80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 >> 172.21.15.29:0/1588893936 conn(0x7f0fe0006180 msgr2=0x7f0fe003c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 shutdown_connections 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 -- 172.21.15.29:0/1588893936 wait complete. 2023-08-22T23:22:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.927+0000 7f0feed19640 1 librados: shutdown 2023-08-22T23:22:33.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.5 2023-08-22T23:22:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fed9c005580 con 0x7feda40aa1a0 2023-08-22T23:22:33.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb701d9b640 1 -- 172.21.15.29:0/947497573 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec0317a0 con 0x7fb7040add90 2023-08-22T23:22:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb701d9b640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb701d9b640 1 -- 172.21.15.29:0/947497573 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec031b30 con 0x7fb7040add90 2023-08-22T23:22:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb701d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb701d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb701d9b640 10 monclient: dump: 2023-08-22T23:22:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:33.981 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb70895d640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:33.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fb70895d640 4 set_mon_vals no callback set 2023-08-22T23:22:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7fed9affd640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed8806ba60 0x7fed8806df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7feda9852640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed8806ba60 0x7fed8806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.975+0000 7feda9852640 10 monclient: get_auth_request con 0x7fed8806ba60 auth_method 0 2023-08-22T23:22:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 10 monclient: shutdown 2023-08-22T23:22:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 -- 172.21.15.29:0/947497573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 msgr2=0x7fb70413e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 --2- 172.21.15.29:0/947497573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 0x7fb70413e190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009870 tx=0x7fb6ec02b8f0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 -- 172.21.15.29:0/947497573 shutdown_connections 2023-08-22T23:22:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 --2- 172.21.15.29:0/947497573 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb70414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 --2- 172.21.15.29:0/947497573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 0x7fb70413e190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 -- 172.21.15.29:0/947497573 >> 172.21.15.29:0/947497573 conn(0x7fb704065ed0 msgr2=0x7fb7040b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 -- 172.21.15.29:0/947497573 shutdown_connections 2023-08-22T23:22:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fb70895d640 1 -- 172.21.15.29:0/947497573 wait complete. 2023-08-22T23:22:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7feda9852640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed8806ba60 0x7fed8806df20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feda0036000 tx=0x7feda0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:33.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.976+0000 7fed9affd640 10 client.4327.objecter ms_handle_connect 0x7fed8806ba60 2023-08-22T23:22:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 10 monclient: build_initial_monmap 2023-08-22T23:22:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 10 monclient: monmap: 2023-08-22T23:22:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:33.992 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 1 librados: starting msgr at 2023-08-22T23:22:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 1 librados: starting objecter 2023-08-22T23:22:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 1 Processor -- start 2023-08-22T23:22:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 1 -- start start 2023-08-22T23:22:33.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 1 librados: setting wanted keys 2023-08-22T23:22:33.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 1 librados: calling monclient init 2023-08-22T23:22:33.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.977+0000 7fb70895d640 10 monclient: init 2023-08-22T23:22:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:22:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 0x7fb704142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7040add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb704143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): picked mon.noname-c con 0x7fb70413e6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7041448e0 0x7fb7041d9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7041448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041d9810 con 0x7fb7041448e0 2023-08-22T23:22:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040b64d0 con 0x7fb7040add90 2023-08-22T23:22:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040b4750 con 0x7fb70413e6d0 2023-08-22T23:22:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb70895d640 10 monclient(hunting): authenticate will time out at 1518.228638s 2023-08-22T23:22:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fed9c057ba0 con 0x7feda40aa1a0 2023-08-22T23:22:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fed9affd640 10 client.4327.objecter ms_dispatch 0x7feda4000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:34.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fed9affd640 3 client.4327.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:34.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fed9affd640 3 client.4327.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:34.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fed9affd640 20 client.4327.objecter dump_active .. 0 homeless 2023-08-22T23:22:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb702d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7041448e0 0x7fb7041d9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.978+0000 7fb6fbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb704143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb704143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43554/0 (socket says 172.21.15.29:43554) 2023-08-22T23:22:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 1 -- 172.21.15.29:0/890059764 learned_addr learned my addr 172.21.15.29:0/890059764 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7041448e0 0x7fb7041d9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33540/0 (socket says 172.21.15.29:33540) 2023-08-22T23:22:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): get_auth_request con 0x7fb70413e6d0 auth_method 0 2023-08-22T23:22:34.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): get_auth_request con 0x7fb7041448e0 auth_method 0 2023-08-22T23:22:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 1 -- 172.21.15.29:0/2475181746 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939802ca80 con 0x7f93a013e190 2023-08-22T23:22:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: dump: 2023-08-22T23:22:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.009 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 1 -- 172.21.15.29:0/2475181746 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939802cbe0 con 0x7f93a013e190 2023-08-22T23:22:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 1 -- 172.21.15.29:0/2475181746 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93980343f0 con 0x7f93a013e190 2023-08-22T23:22:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f93a6046640 5 monclient: authenticate success, global_id 4324 2023-08-22T23:22:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f93a6046640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f93a6046640 4 set_mon_vals no callback set 2023-08-22T23:22:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7f939e7fc640 10 monclient: dump: 2023-08-22T23:22:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.015 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.979+0000 7fb702d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 10 monclient: shutdown 2023-08-22T23:22:34.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 -- 172.21.15.29:0/2475181746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 msgr2=0x7f93a013e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 --2- 172.21.15.29:0/2475181746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 0x7f93a013e610 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f93a00ad970 tx=0x7f939802bee0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 -- 172.21.15.29:0/2475181746 shutdown_connections 2023-08-22T23:22:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 --2- 172.21.15.29:0/2475181746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 0x7f93a013e610 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 --2- 172.21.15.29:0/2475181746 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a0145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 --2- 172.21.15.29:0/2475181746 >> v2:172.21.15.29:3300/0 conn(0x7f93a01448e0 0x7f93a0144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 -- 172.21.15.29:0/2475181746 >> 172.21.15.29:0/2475181746 conn(0x7f93a0065ed0 msgr2=0x7f93a00b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 -- 172.21.15.29:0/2475181746 shutdown_connections 2023-08-22T23:22:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.980+0000 7f93a6046640 1 -- 172.21.15.29:0/2475181746 wait complete. 2023-08-22T23:22:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 10 monclient: build_initial_monmap 2023-08-22T23:22:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 10 monclient: monmap: 2023-08-22T23:22:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:34.026 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 1 librados: starting msgr at 2023-08-22T23:22:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 1 librados: starting objecter 2023-08-22T23:22:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 1 Processor -- start 2023-08-22T23:22:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 1 -- start start 2023-08-22T23:22:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 1 librados: setting wanted keys 2023-08-22T23:22:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 1 librados: calling monclient init 2023-08-22T23:22:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.981+0000 7f93a6046640 10 monclient: init 2023-08-22T23:22:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:22:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a00a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): picked mon.noname-c con 0x7f93a00add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 0x7f93a0141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): picked mon.noname-b con 0x7f93a013e190 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 0x7f93a01d6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): picked mon.noname-a con 0x7f93a01448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00a84a0 con 0x7f93a01448e0 2023-08-22T23:22:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0140b90 con 0x7f93a013e190 2023-08-22T23:22:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a01d6220 con 0x7f93a00add90 2023-08-22T23:22:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7f93a6046640 10 monclient(hunting): authenticate will time out at 1518.232931s 2023-08-22T23:22:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-08-22T23:22:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 1 -- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 msgr2=0x7fb704142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 0x7fb704142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 1 -- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7041448e0 msgr2=0x7fb7041d9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7041448e0 0x7fb7041d9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 1 monclient: found mon.noname-c 2023-08-22T23:22:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.982+0000 7fb6fbfff640 1 -- 172.21.15.29:0/890059764 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec0094d0 con 0x7fb70413e6d0 2023-08-22T23:22:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.985+0000 7fb6fbfff640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb704143ae0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009870 tx=0x7fb6ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.985+0000 7fb6faffd640 10 client.?.objecter ms_handle_connect 0x7fb70413e6d0 2023-08-22T23:22:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.985+0000 7fb6faffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 0x7f93a01d6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 0x7f93a01d6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33550/0 (socket says 172.21.15.29:33550) 2023-08-22T23:22:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 1 -- 172.21.15.29:0/785842337 learned_addr learned my addr 172.21.15.29:0/785842337 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 10 monclient(hunting): get_auth_request con 0x7f93a01448e0 auth_method 0 2023-08-22T23:22:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.986+0000 7f939ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.987+0000 7f939ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.987+0000 7f939ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.987+0000 7f939ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.988+0000 7f939f7fe640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a00a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec03a930 con 0x7fb70413e6d0 2023-08-22T23:22:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: dump: 2023-08-22T23:22:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec03aa90 con 0x7fb70413e6d0 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb70895d640 5 monclient: authenticate success, global_id 4295 2023-08-22T23:22:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb70895d640 10 monclient: _renew_subs 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb70895d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb704144020 con 0x7fb70413e6d0 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec03ae40 con 0x7fb70413e6d0 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb6faffd640 10 monclient: dump: 2023-08-22T23:22:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.046 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7f939f7fe640 10 monclient(hunting): get_auth_request con 0x7f93a00add90 auth_method 0 2023-08-22T23:22:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7f939f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7f939f7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7f939f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7fb70895d640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7fb70895d640 10 monclient: _renew_subs 2023-08-22T23:22:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7fb70895d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7041da1a0 con 0x7fb70413e6d0 2023-08-22T23:22:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.989+0000 7fb70159a640 4 set_mon_vals no callback set 2023-08-22T23:22:34.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.990+0000 7fb70895d640 1 librados: init done 2023-08-22T23:22:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2023-08-22T23:22:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 1 -- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 msgr2=0x7f93a00a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a00a7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 1 -- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 msgr2=0x7f93a0141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 0x7f93a0141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 1 monclient: found mon.noname-a 2023-08-22T23:22:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7f939ffff640 1 -- 172.21.15.29:0/785842337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398009b30 con 0x7f93a01448e0 2023-08-22T23:22:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fb70895d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fb70895d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fb70895d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7040b0360 con 0x7fb70413e6d0 2023-08-22T23:22:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fed9c053850 con 0x7feda40aa1a0 2023-08-22T23:22:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fed9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.991+0000 7fed9affd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.993+0000 7f939ffff640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 0x7f93a01d6740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9398036040 tx=0x7f9398004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.993+0000 7f9387fff640 10 client.?.objecter ms_handle_connect 0x7f93a01448e0 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.993+0000 7f9387fff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939803e450 con 0x7f93a01448e0 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: dump: 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: _finish_auth 0 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: _check_auth_tickets 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939803e5d0 con 0x7f93a01448e0 2023-08-22T23:22:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 5 monclient: authenticate success, global_id 4329 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 10 monclient: _renew_subs 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a00a7de0 con 0x7f93a01448e0 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939803e980 con 0x7f93a01448e0 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f9387fff640 10 monclient: dump: 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f939dffb640 4 set_mon_vals no callback set 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 10 monclient: _renew_subs 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a013f0d0 con 0x7f93a01448e0 2023-08-22T23:22:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.998+0000 7f93a6046640 1 librados: init done 2023-08-22T23:22:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.999+0000 7f93a6046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.999+0000 7f93a6046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.999+0000 7f93a6046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:33.999+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a0066cb0 con 0x7f93a01448e0 2023-08-22T23:22:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.019+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb6ec02f050 con 0x7fb70413e6d0 2023-08-22T23:22:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.019+0000 7fb6faffd640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb6e406ba30 0x7fb6e406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.020+0000 7fb70259c640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb6e406ba30 0x7fb6e406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.020+0000 7fb70259c640 10 monclient: get_auth_request con 0x7fb6e406ba30 auth_method 0 2023-08-22T23:22:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.020+0000 7fb70259c640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb6e406ba30 0x7fb6e406def0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009650 tx=0x7fb6f4006ba0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.020+0000 7fb6faffd640 10 client.4295.objecter ms_handle_connect 0x7fb6e406ba30 2023-08-22T23:22:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.022+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f939802cd50 con 0x7f93a01448e0 2023-08-22T23:22:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.023+0000 7f9387fff640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f938c06ba30 0x7f938c06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.023+0000 7f939f7fe640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f938c06ba30 0x7f938c06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.023+0000 7f939f7fe640 10 monclient: get_auth_request con 0x7f938c06ba30 auth_method 0 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.023+0000 7f939f7fe640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f938c06ba30 0x7f938c06def0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f939000b810 tx=0x7f9390008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.023+0000 7f9387fff640 10 client.4329.objecter ms_handle_connect 0x7f938c06ba30 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.025+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f939807b380 con 0x7f93a01448e0 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.025+0000 7f9387fff640 10 client.4329.objecter ms_dispatch 0x7f93a0000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.025+0000 7f9387fff640 3 client.4329.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.025+0000 7f9387fff640 3 client.4329.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.025+0000 7f9387fff640 20 client.4329.objecter dump_active .. 0 homeless 2023-08-22T23:22:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.054+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0cc0014440 con 0x7f0cc8142af0 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.055+0000 7f0cbd7fa640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0cb406ba10 0x7f0cb406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.055+0000 7f0cccf27640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0cb406ba10 0x7f0cb406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.055+0000 7f0cccf27640 10 monclient: get_auth_request con 0x7f0cb406ba10 auth_method 0 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.056+0000 7f0cbd7fa640 10 client.4333.objecter ms_handle_connect 0x7f0cb406ba10 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.056+0000 7f0cccf27640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0cb406ba10 0x7f0cb406ded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009990 tx=0x7f0cc4009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.061+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0cc0057fb0 con 0x7f0cc8142af0 2023-08-22T23:22:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.061+0000 7f0cbd7fa640 10 client.4333.objecter ms_dispatch 0x7f0cc8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.061+0000 7f0cbd7fa640 3 client.4333.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.061+0000 7f0cbd7fa640 3 client.4333.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.061+0000 7f0cbd7fa640 20 client.4333.objecter dump_active .. 0 homeless 2023-08-22T23:22:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.073+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f0cc0057d80 con 0x7f0cc8142af0 2023-08-22T23:22:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.073+0000 7f0cbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.073+0000 7f0cbd7fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.095+0000 7f9768ff9640 1 -- 172.21.15.29:0/2766711473 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9764043020 con 0x7f976c0713c0 2023-08-22T23:22:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.095+0000 7f9768ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-08-22T23:22:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.095+0000 7f9768ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:34.098 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-08-22T23:22:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.098+0000 7f9772807640 10 librados: watch_flush enter 2023-08-22T23:22:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.098+0000 7f9772807640 10 librados: watch_flush exit 2023-08-22T23:22:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.098+0000 7f9772807640 20 client.4330.objecter shutdown clearing up homeless session... 2023-08-22T23:22:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.098+0000 7f9772807640 10 client.4330.objecter successfully canceled tick 2023-08-22T23:22:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f975806b9e0 msgr2=0x7f975806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f975806b9e0 0x7f975806dea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9760006fd0 tx=0x7f9760008040 comp rx=0 tx=0).stop 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 10 monclient: shutdown 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 msgr2=0x7f976c147d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 0x7f976c147d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9764036040 tx=0x7f9764004a40 comp rx=0 tx=0).stop 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 shutdown_connections 2023-08-22T23:22:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f975806b9e0 0x7f975806dea0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f976c0adfc0 0x7f976c0a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f976c0a8a10 0x7f976c0a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 --2- 172.21.15.29:0/2766711473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f976c0713c0 0x7f976c147d40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 >> 172.21.15.29:0/2766711473 conn(0x7f976c0aee40 msgr2=0x7f976c073190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 shutdown_connections 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 -- 172.21.15.29:0/2766711473 wait complete. 2023-08-22T23:22:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.099+0000 7f9772807640 1 librados: shutdown 2023-08-22T23:22:34.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.7 2023-08-22T23:22:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.149+0000 7f07fd6b5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-08-22T23:22:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.149+0000 7f07fd6b5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-08-22T23:22:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.149+0000 7f07fd6b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.149+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f07f813f640 con 0x7f07f80add90 2023-08-22T23:22:34.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.195+0000 7fedac2b7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-08-22T23:22:34.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.195+0000 7fedac2b7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-08-22T23:22:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.195+0000 7fedac2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.195+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7feda41ddef0 con 0x7feda40aa1a0 2023-08-22T23:22:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb6ec07a3d0 con 0x7fb70413e6d0 2023-08-22T23:22:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 10 client.4295.objecter ms_dispatch 0x7fb704000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 3 client.4295.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 3 client.4295.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 20 client.4295.objecter dump_active .. 0 homeless 2023-08-22T23:22:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fb6ec076080 con 0x7fb70413e6d0 2023-08-22T23:22:34.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.307+0000 7fb6faffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.316+0000 7f07ef7fe640 1 -- 172.21.15.29:0/2182184821 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07f813f640 con 0x7f07f80add90 2023-08-22T23:22:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.316+0000 7f07ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-08-22T23:22:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.316+0000 7f07ef7fe640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:34.318 INFO:teuthology.orchestra.run.smithi029.stdout:68719476740 2023-08-22T23:22:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 10 librados: watch_flush enter 2023-08-22T23:22:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 10 librados: watch_flush exit 2023-08-22T23:22:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 20 client.4292.objecter shutdown clearing up homeless session... 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 10 client.4292.objecter successfully canceled tick 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f07d806b9e0 msgr2=0x7f07d806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f07d806b9e0 0x7f07d806dea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f07f8137950 tx=0x7f07e0007930 comp rx=0 tx=0).stop 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 10 monclient: shutdown 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 msgr2=0x7f07f8135e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 0x7f07f8135e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009890 tx=0x7f07e80046c0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 shutdown_connections 2023-08-22T23:22:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f07d806b9e0 0x7f07d806dea0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f07f81448e0 0x7f07f8132a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f813e8a0 0x7f07f8136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 --2- 172.21.15.29:0/2182184821 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f07f80add90 0x7f07f8135e60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 >> 172.21.15.29:0/2182184821 conn(0x7f07f8065ed0 msgr2=0x7f07f8144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.318+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 shutdown_connections 2023-08-22T23:22:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.319+0000 7f07fd6b5640 1 -- 172.21.15.29:0/2182184821 wait complete. 2023-08-22T23:22:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.319+0000 7f07fd6b5640 1 librados: shutdown 2023-08-22T23:22:34.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.323+0000 7f0ccdf29640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-08-22T23:22:34.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.323+0000 7f0ccdf29640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-08-22T23:22:34.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.323+0000 7f0ccdf29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:34.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.323+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0cc81407c0 con 0x7f0cc8142af0 2023-08-22T23:22:34.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.0 2023-08-22T23:22:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.386+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f9398077030 con 0x7f93a01448e0 2023-08-22T23:22:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.387+0000 7f9387fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:34.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.387+0000 7f9387fff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:34.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.426+0000 7fed9affd640 1 -- 172.21.15.29:0/186695482 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feda41ddef0 con 0x7feda40aa1a0 2023-08-22T23:22:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.426+0000 7fed9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-08-22T23:22:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.426+0000 7fed9affd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:34.429 INFO:teuthology.orchestra.run.smithi029.stdout:64424509445 2023-08-22T23:22:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.427+0000 7f0cbd7fa640 1 -- 172.21.15.29:0/2051262038 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0cc0019020 con 0x7f0cc8142af0 2023-08-22T23:22:34.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.427+0000 7f0cbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-08-22T23:22:34.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.427+0000 7f0cbd7fa640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:34.430 INFO:teuthology.orchestra.run.smithi029.stdout:68719476740 2023-08-22T23:22:34.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 10 librados: watch_flush enter 2023-08-22T23:22:34.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 10 librados: watch_flush exit 2023-08-22T23:22:34.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 20 client.4327.objecter shutdown clearing up homeless session... 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 10 client.4327.objecter successfully canceled tick 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed8806ba60 msgr2=0x7fed8806df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed8806ba60 0x7fed8806df20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feda0036000 tx=0x7feda0030000 comp rx=0 tx=0).stop 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 10 monclient: shutdown 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 msgr2=0x7feda406d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.429+0000 7fedac2b7640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 0x7feda406d940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00c800 tx=0x7fed9c00ccc0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 shutdown_connections 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed8806ba60 0x7fed8806df20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7feda4149380 0x7feda41de1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40aafd0 0x7feda406e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 --2- 172.21.15.29:0/186695482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feda40aa1a0 0x7feda406d940 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 >> 172.21.15.29:0/186695482 conn(0x7feda40b27e0 msgr2=0x7feda4148aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 shutdown_connections 2023-08-22T23:22:34.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 -- 172.21.15.29:0/186695482 wait complete. 2023-08-22T23:22:34.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7fedac2b7640 1 librados: shutdown 2023-08-22T23:22:34.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7f0ccdf29640 10 librados: watch_flush enter 2023-08-22T23:22:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7f0ccdf29640 10 librados: watch_flush exit 2023-08-22T23:22:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7f0ccdf29640 20 client.4333.objecter shutdown clearing up homeless session... 2023-08-22T23:22:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.430+0000 7f0ccdf29640 10 client.4333.objecter successfully canceled tick 2023-08-22T23:22:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0cb406ba10 msgr2=0x7f0cb406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0cb406ba10 0x7f0cb406ded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009990 tx=0x7f0cc4009110 comp rx=0 tx=0).stop 2023-08-22T23:22:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 10 monclient: shutdown 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 msgr2=0x7f0cc80aa510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 0x7f0cc80aa510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000b990 tx=0x7f0cc000bd90 comp rx=0 tx=0).stop 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 shutdown_connections 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0cb406ba10 0x7f0cb406ded0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cc8142af0 0x7f0cc80aa510 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc813f460 0x7f0cc81da7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 --2- 172.21.15.29:0/2051262038 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0cc8070350 0x7f0cc8140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 >> 172.21.15.29:0/2051262038 conn(0x7f0cc80b4640 msgr2=0x7f0cc80b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 shutdown_connections 2023-08-22T23:22:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 -- 172.21.15.29:0/2051262038 wait complete. 2023-08-22T23:22:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.431+0000 7f0ccdf29640 1 librados: shutdown 2023-08-22T23:22:34.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.6 2023-08-22T23:22:34.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.2 2023-08-22T23:22:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.514+0000 7fb70895d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-08-22T23:22:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.514+0000 7fb70895d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-08-22T23:22:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.514+0000 7fb70895d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.514+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb7040b0360 con 0x7fb70413e6d0 2023-08-22T23:22:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.551+0000 7fb6faffd640 1 -- 172.21.15.29:0/890059764 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb7040b0360 con 0x7fb70413e6d0 2023-08-22T23:22:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.551+0000 7fb6faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-08-22T23:22:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.551+0000 7fb6faffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:34.554 INFO:teuthology.orchestra.run.smithi029.stdout:68719476740 2023-08-22T23:22:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 10 librados: watch_flush enter 2023-08-22T23:22:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 10 librados: watch_flush exit 2023-08-22T23:22:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 20 client.4295.objecter shutdown clearing up homeless session... 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 10 client.4295.objecter successfully canceled tick 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb6e406ba30 msgr2=0x7fb6e406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb6e406ba30 0x7fb6e406def0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009650 tx=0x7fb6f4006ba0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 10 monclient: shutdown 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 msgr2=0x7fb704143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.553+0000 7fb70895d640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb704143ae0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009870 tx=0x7fb6ec0046c0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 shutdown_connections 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb6e406ba30 0x7fb6e406def0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7041448e0 0x7fb7041d9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb70413e6d0 0x7fb704143ae0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 --2- 172.21.15.29:0/890059764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb7040add90 0x7fb704142ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 >> 172.21.15.29:0/890059764 conn(0x7fb704065ed0 msgr2=0x7fb704138010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 shutdown_connections 2023-08-22T23:22:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 -- 172.21.15.29:0/890059764 wait complete. 2023-08-22T23:22:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.554+0000 7fb70895d640 1 librados: shutdown 2023-08-22T23:22:34.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.1 2023-08-22T23:22:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.590+0000 7f93a6046640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-08-22T23:22:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.590+0000 7f93a6046640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-08-22T23:22:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.590+0000 7f93a6046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.590+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f93a01d63e0 con 0x7f93a01448e0 2023-08-22T23:22:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.606+0000 7f9387fff640 1 -- 172.21.15.29:0/785842337 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9398043020 con 0x7f93a01448e0 2023-08-22T23:22:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.606+0000 7f9387fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-08-22T23:22:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.606+0000 7f9387fff640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:34.609 INFO:teuthology.orchestra.run.smithi029.stdout:68719476743 2023-08-22T23:22:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 10 librados: watch_flush enter 2023-08-22T23:22:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 10 librados: watch_flush exit 2023-08-22T23:22:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 20 client.4329.objecter shutdown clearing up homeless session... 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 10 client.4329.objecter successfully canceled tick 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f938c06ba30 msgr2=0x7f938c06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f938c06ba30 0x7f938c06def0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f939000b810 tx=0x7f9390008040 comp rx=0 tx=0).stop 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 10 monclient: shutdown 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 msgr2=0x7f93a01d6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 0x7f93a01d6740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9398036040 tx=0x7f9398004a40 comp rx=0 tx=0).stop 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 shutdown_connections 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f938c06ba30 0x7f938c06def0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93a01448e0 0x7f93a01d6740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93a013e190 0x7f93a0141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 --2- 172.21.15.29:0/785842337 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f93a00add90 0x7f93a00a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.608+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 >> 172.21.15.29:0/785842337 conn(0x7f93a0065ed0 msgr2=0x7f93a0144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.609+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 shutdown_connections 2023-08-22T23:22:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.609+0000 7f93a6046640 1 -- 172.21.15.29:0/785842337 wait complete. 2023-08-22T23:22:34.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.609+0000 7f93a6046640 1 librados: shutdown 2023-08-22T23:22:34.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476743 for osd.3 2023-08-22T23:22:34.622 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:34.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-08-22T23:22:34.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-08-22T23:22:34.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.968+0000 7fc0c7c83640 10 monclient: get_monmap_and_config 2023-08-22T23:22:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.968+0000 7fc0c7c83640 10 monclient: build_initial_monmap 2023-08-22T23:22:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.968+0000 7fc0c7c83640 10 monclient: monmap: 2023-08-22T23:22:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:34.969517+0000 2023-08-22T23:22:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:34.969517+0000 2023-08-22T23:22:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:34.972 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.968+0000 7fc0c7c83640 1 Processor -- start 2023-08-22T23:22:34.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 1 -- start start 2023-08-22T23:22:34.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient: init 2023-08-22T23:22:34.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0c01448e0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0c00add90 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fc0c013e8a0 0x7fc0c014ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c013e8a0 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00b12d0 con 0x7fc0c00add90 2023-08-22T23:22:34.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00afae0 con 0x7fc0c01448e0 2023-08-22T23:22:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c00afdc0 con 0x7fc0c013e8a0 2023-08-22T23:22:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.969+0000 7fc0c7c83640 10 monclient(hunting): authenticate will time out at 1519.220193s 2023-08-22T23:22:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.970+0000 7fc0c521e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc0c00add90 0x7fc0c013e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59990/0 (socket says 172.21.15.29:59990) 2023-08-22T23:22:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.970+0000 7fc0c521e640 1 -- 172.21.15.29:0/1120939659 learned_addr learned my addr 172.21.15.29:0/1120939659 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.970+0000 7fc0c6220640 1 --2- 172.21.15.29:0/1120939659 >> v2:172.21.15.52:3300/0 conn(0x7fc0c013e8a0 0x7fc0c014ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.971+0000 7fc0c6220640 10 monclient(hunting): get_auth_request con 0x7fc0c013e8a0 auth_method 0 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.971+0000 7fc0c6220640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.971+0000 7fc0c6220640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.971+0000 7fc0c6220640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c6220640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:34.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c6220640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c6220640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c4a1d640 1 -- 172.21.15.29:0/1120939659 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424323615 0 0) 0x7fc0c00b12d0 con 0x7fc0c00add90 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c4a1d640 10 monclient(hunting): _negotiate 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c4a1d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c4a1d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c4a1d640 10 monclient(hunting): my global_id is 4332 2023-08-22T23:22:34.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.972+0000 7fc0c4a1d640 1 -- 172.21.15.29:0/1120939659 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac003520 con 0x7fc0c00add90 2023-08-22T23:22:34.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2023-08-22T23:22:34.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 1 -- 172.21.15.29:0/1120939659 >> v1:172.21.15.52:6789/0 conn(0x7fc0c01448e0 legacy=0x7fc0c0144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 1 -- 172.21.15.29:0/1120939659 >> v1:172.21.15.29:6789/0 conn(0x7fc0c00add90 legacy=0x7fc0c013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 10 monclient: _finish_hunting 0 2023-08-22T23:22:34.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 1 monclient: found mon.noname-c 2023-08-22T23:22:34.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:34.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:34.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.975+0000 7fc0c6220640 1 -- 172.21.15.29:0/1120939659 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00b00a0 con 0x7fc0c013e8a0 2023-08-22T23:22:34.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.979+0000 7fc0c6220640 1 --2- 172.21.15.29:0/1120939659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c014ca10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009d90 tx=0x7fc0b802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0b6ccd27fc137 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:34.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.982+0000 7fc0c4a1d640 1 -- 172.21.15.29:0/1120939659 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b802cc60 con 0x7fc0c013e8a0 2023-08-22T23:22:34.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.982+0000 7fc0c4a1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.982+0000 7fc0c4a1d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.982+0000 7fc0c4a1d640 10 monclient: dump: 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 10 monclient: _finish_auth 0 2023-08-22T23:22:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 10 monclient: _check_auth_tickets 2023-08-22T23:22:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 1 -- 172.21.15.29:0/1120939659 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b802cdc0 con 0x7fc0c013e8a0 2023-08-22T23:22:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 5 monclient: authenticate success, global_id 4342 2023-08-22T23:22:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 1 -- 172.21.15.29:0/1120939659 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8034660 con 0x7fc0c013e8a0 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 4 set_mon_vals no callback set 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c4a1d640 10 monclient: dump: 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:34.990 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 10 monclient: shutdown 2023-08-22T23:22:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 -- 172.21.15.29:0/1120939659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 msgr2=0x7fc0c014ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/1120939659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c014ca10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009d90 tx=0x7fc0b802beb0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 -- 172.21.15.29:0/1120939659 shutdown_connections 2023-08-22T23:22:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/1120939659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c014ca10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 -- 172.21.15.29:0/1120939659 >> 172.21.15.29:0/1120939659 conn(0x7fc0c0065ed0 msgr2=0x7fc0c00b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 -- 172.21.15.29:0/1120939659 shutdown_connections 2023-08-22T23:22:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.983+0000 7fc0c7c83640 1 -- 172.21.15.29:0/1120939659 wait complete. 2023-08-22T23:22:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 10 monclient: build_initial_monmap 2023-08-22T23:22:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 10 monclient: monmap: 2023-08-22T23:22:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 1 librados: starting msgr at 2023-08-22T23:22:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 1 librados: starting objecter 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 1 Processor -- start 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 1 -- start start 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 1 librados: setting wanted keys 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 1 librados: calling monclient init 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.984+0000 7fc0c7c83640 10 monclient: init 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:22:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc0c00add90 0x7fc0c0143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c00add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c01d9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c013e8a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c01448e0 0x7fc0c01d5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c01448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c01d9ec0 con 0x7fc0c01448e0 2023-08-22T23:22:34.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c01d9aa0 con 0x7fc0c013e8a0 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0144020 con 0x7fc0c00add90 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.985+0000 7fc0c7c83640 10 monclient(hunting): authenticate will time out at 1519.235976s 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c01d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c5a1f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc0c00add90 0x7fc0c0143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c6220640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c01448e0 0x7fc0c01d5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c01d9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39746/0 (socket says 172.21.15.29:39746) 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 1 -- 172.21.15.29:0/2003751952 learned_addr learned my addr 172.21.15.29:0/2003751952 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): get_auth_request con 0x7fc0c013e8a0 auth_method 0 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c6220640 10 monclient(hunting): get_auth_request con 0x7fc0c01448e0 auth_method 0 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c6220640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c6220640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c6220640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c5a1f640 10 monclient(hunting): get_auth_request con 0x7fc0c00add90 auth_method 0 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c5a1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c5a1f640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c5a1f640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.986+0000 7fc0c521e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.987+0000 7fc0c6220640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.987+0000 7fc0c6220640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.987+0000 7fc0c6220640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.987+0000 7fc0c5a1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.987+0000 7fc0c5a1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.987+0000 7fc0c5a1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 1 -- 172.21.15.29:0/2003751952 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc0c00add90 msgr2=0x7fc0c0143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc0c00add90 0x7fc0c0143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 1 -- 172.21.15.29:0/2003751952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c01448e0 msgr2=0x7fc0c01d5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c01448e0 0x7fc0c01d5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 1 monclient: found mon.noname-b 2023-08-22T23:22:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.989+0000 7fc0c521e640 1 -- 172.21.15.29:0/2003751952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b8009a70 con 0x7fc0c013e8a0 2023-08-22T23:22:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.992+0000 7fc0c521e640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c01d9560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c500 tx=0x7fc0b000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.992+0000 7fc0b67fc640 10 client.?.objecter ms_handle_connect 0x7fc0c013e8a0 2023-08-22T23:22:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.992+0000 7fc0b67fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:35.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0005ce0 con 0x7fc0c013e8a0 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: dump: 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.008 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b0005e40 con 0x7fc0c013e8a0 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 5 monclient: authenticate success, global_id 4348 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 10 monclient: _renew_subs 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c01da2e0 con 0x7fc0c013e8a0 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0007330 con 0x7fc0c013e8a0 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b67fc640 10 monclient: dump: 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.011 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0b7fff640 4 set_mon_vals no callback set 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 10 monclient: _renew_subs 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c01d6700 con 0x7fc0c013e8a0 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.996+0000 7fc0c7c83640 1 librados: init done 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.997+0000 7fc0c7c83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.997+0000 7fc0c7c83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.997+0000 7fc0c7c83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:34.997+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c00a8250 con 0x7fc0c013e8a0 2023-08-22T23:22:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.054+0000 7ff0c8fc4640 10 monclient: get_monmap_and_config 2023-08-22T23:22:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.054+0000 7ff0c8fc4640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.054+0000 7ff0c8fc4640 10 monclient: monmap: 2023-08-22T23:22:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:35.056095+0000 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:35.056095+0000 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:35.058 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.055+0000 7ff0c8fc4640 1 Processor -- start 2023-08-22T23:22:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.055+0000 7ff0c8fc4640 1 -- start start 2023-08-22T23:22:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.055+0000 7ff0c8fc4640 10 monclient: init 2023-08-22T23:22:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0c41448e0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ff0c40add90 0x7ff0c413e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c40add90 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 0x7ff0c414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c413e6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40b12d0 con 0x7ff0c41448e0 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40afae0 con 0x7ff0c40add90 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40afdc0 con 0x7ff0c413e6d0 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c8fc4640 10 monclient(hunting): authenticate will time out at 1519.306747s 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.056+0000 7ff0c2d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 0x7ff0c414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 0x7ff0c414ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43590/0 (socket says 172.21.15.29:43590) 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1316807070 learned_addr learned my addr 172.21.15.29:0/1316807070 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): get_auth_request con 0x7ff0c413e6d0 auth_method 0 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.057+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2023-08-22T23:22:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1316807070 >> v2:172.21.15.52:3300/0 conn(0x7ff0c40add90 msgr2=0x7ff0c413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 1 --2- 172.21.15.29:0/1316807070 >> v2:172.21.15.52:3300/0 conn(0x7ff0c40add90 0x7ff0c413e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1316807070 >> v1:172.21.15.52:6789/0 conn(0x7ff0c41448e0 legacy=0x7ff0c4144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 1 monclient: found mon.noname-b 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.059+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1316807070 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40b00a0 con 0x7ff0c413e6d0 2023-08-22T23:22:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.062+0000 7ff0c2d9d640 1 --2- 172.21.15.29:0/1316807070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 0x7ff0c414ba00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009e50 tx=0x7ff0bc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c26ba123c69516ad server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 1 -- 172.21.15.29:0/1316807070 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc02ca70 con 0x7ff0c413e6d0 2023-08-22T23:22:35.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:35.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: dump: 2023-08-22T23:22:35.068 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.068 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.069 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 1 -- 172.21.15.29:0/1316807070 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc02cbd0 con 0x7ff0c413e6d0 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c8fc4640 5 monclient: authenticate success, global_id 4307 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c8fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c8fc4640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c8fc4640 4 set_mon_vals no callback set 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 1 -- 172.21.15.29:0/1316807070 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0343f0 con 0x7ff0c413e6d0 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.066+0000 7ff0c159a640 10 monclient: dump: 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.071 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.072 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 10 monclient: shutdown 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1316807070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 msgr2=0x7ff0c414ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1316807070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 0x7ff0c414ba00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009e50 tx=0x7ff0bc02bed0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1316807070 shutdown_connections 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1316807070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c413e6d0 0x7ff0c414ba00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1316807070 >> v2:172.21.15.52:3300/0 conn(0x7ff0c40add90 0x7ff0c413e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1316807070 >> 172.21.15.29:0/1316807070 conn(0x7ff0c4065ed0 msgr2=0x7ff0c40b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1316807070 shutdown_connections 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.067+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1316807070 wait complete. 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.068+0000 7ff0c8fc4640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.068+0000 7ff0c8fc4640 10 monclient: monmap: 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.068+0000 7ff0c8fc4640 1 librados: starting msgr at 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.068+0000 7ff0c8fc4640 1 librados: starting objecter 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 Processor -- start 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 -- start start 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 librados: setting wanted keys 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 librados: calling monclient init 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient: init 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c40add90 0x7ff0c4144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c40add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff0c413e6d0 0x7ff0c41412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c413e6d0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 0x7ff0c4141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c41448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4141d50 con 0x7ff0c41448e0 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c413f2d0 con 0x7ff0c413e6d0 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c413eef0 con 0x7ff0c40add90 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.069+0000 7ff0c8fc4640 10 monclient(hunting): authenticate will time out at 1519.320135s 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c259c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c40add90 0x7ff0c4144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 0x7ff0c4141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 0x7ff0c4141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33572/0 (socket says 172.21.15.29:33572) 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1375339479 learned_addr learned my addr 172.21.15.29:0/1375339479 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 10 monclient(hunting): get_auth_request con 0x7ff0c41448e0 auth_method 0 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c259c640 10 monclient(hunting): get_auth_request con 0x7ff0c40add90 auth_method 0 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c259c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.070+0000 7ff0c259c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.071+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.071+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.071+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.071+0000 7ff0c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.071+0000 7ff0c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.071+0000 7ff0c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.073+0000 7ff0c2d9d640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2023-08-22T23:22:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.073+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c40add90 msgr2=0x7ff0c4144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.073+0000 7ff0c2d9d640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c40add90 0x7ff0c4144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.073+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff0c413e6d0 msgr2=0x7ff0c41412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.073+0000 7ff0c2d9d640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff0c413e6d0 0x7ff0c41412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.074+0000 7ff0c2d9d640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.074+0000 7ff0c2d9d640 1 monclient: found mon.noname-a 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.074+0000 7ff0c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.074+0000 7ff0c2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.074+0000 7ff0c2d9d640 1 -- 172.21.15.29:0/1375339479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc009b30 con 0x7ff0c41448e0 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.076+0000 7ff0c2d9d640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 0x7ff0c4141810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc036040 tx=0x7ff0bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.076+0000 7ff0baffd640 10 client.?.objecter ms_handle_connect 0x7ff0c41448e0 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.076+0000 7ff0baffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc03e450 con 0x7ff0c41448e0 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: dump: 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc03e5d0 con 0x7ff0c41448e0 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc03e9c0 con 0x7ff0c41448e0 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 5 monclient: authenticate success, global_id 4338 2023-08-22T23:22:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 10 monclient: _renew_subs 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c413f7f0 con 0x7ff0c41448e0 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c0d99640 4 set_mon_vals no callback set 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0baffd640 10 monclient: dump: 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 10 monclient: _renew_subs 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c413fd70 con 0x7ff0c41448e0 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.080+0000 7ff0c8fc4640 1 librados: init done 2023-08-22T23:22:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.081+0000 7ff0c8fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.081+0000 7ff0c8fc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.081+0000 7ff0c8fc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.081+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c4066cb0 con 0x7ff0c41448e0 2023-08-22T23:22:35.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-08-22T23:22:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.228+0000 7f570168c640 10 monclient: get_monmap_and_config 2023-08-22T23:22:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.228+0000 7f570168c640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.228+0000 7f570168c640 10 monclient: monmap: 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:35.230150+0000 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:35.230150+0000 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:35.232 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.229+0000 7f570168c640 1 Processor -- start 2023-08-22T23:22:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.229+0000 7f570168c640 1 -- start start 2023-08-22T23:22:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.229+0000 7f570168c640 10 monclient: init 2023-08-22T23:22:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): picked mon.noname-b con 0x7f56fc1448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f56fc0add90 0x7f56fc145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): picked mon.noname-c con 0x7f56fc0add90 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56fc13e190 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56fc0b12d0 con 0x7f56fc13e190 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56fc0afae0 con 0x7f56fc0add90 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc0afdc0 con 0x7f56fc1448e0 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.230+0000 7f570168c640 10 monclient(hunting): authenticate will time out at 1519.481119s 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43594/0 (socket says 172.21.15.29:43594) 2023-08-22T23:22:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 1 -- 172.21.15.29:0/2914923014 learned_addr learned my addr 172.21.15.29:0/2914923014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 10 monclient(hunting): get_auth_request con 0x7f56fc1448e0 auth_method 0 2023-08-22T23:22:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.231+0000 7f56faffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.232+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.232+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.232+0000 7f56faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2023-08-22T23:22:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 1 -- 172.21.15.29:0/2914923014 >> v2:172.21.15.52:3300/0 conn(0x7f56fc0add90 msgr2=0x7f56fc145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 1 --2- 172.21.15.29:0/2914923014 >> v2:172.21.15.52:3300/0 conn(0x7f56fc0add90 0x7f56fc145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 1 -- 172.21.15.29:0/2914923014 >> v1:172.21.15.52:6789/0 conn(0x7f56fc13e190 legacy=0x7f56fc1410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 1 monclient: found mon.noname-b 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.234+0000 7f56faffd640 1 -- 172.21.15.29:0/2914923014 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0b00a0 con 0x7f56fc1448e0 2023-08-22T23:22:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.237+0000 7f56faffd640 1 --2- 172.21.15.29:0/2914923014 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc144cc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009870 tx=0x7f56ec02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fbf5ed89adce522 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 1 -- 172.21.15.29:0/2914923014 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec031640 con 0x7f56fc1448e0 2023-08-22T23:22:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 10 monclient: dump: 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 1 -- 172.21.15.29:0/2914923014 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec0317a0 con 0x7f56fc1448e0 2023-08-22T23:22:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f570168c640 5 monclient: authenticate success, global_id 4313 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f570168c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.241+0000 7f56f9ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f56f9ffb640 1 -- 172.21.15.29:0/2914923014 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec031b30 con 0x7f56fc1448e0 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f56f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f56f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f56f9ffb640 10 monclient: dump: 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.247 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:35.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 4 set_mon_vals no callback set 2023-08-22T23:22:35.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 10 monclient: shutdown 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 -- 172.21.15.29:0/2914923014 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 msgr2=0x7f56fc144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 --2- 172.21.15.29:0/2914923014 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc144cc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009870 tx=0x7f56ec02b8f0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 -- 172.21.15.29:0/2914923014 shutdown_connections 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 --2- 172.21.15.29:0/2914923014 >> v2:172.21.15.52:3300/0 conn(0x7f56fc0add90 0x7f56fc145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 --2- 172.21.15.29:0/2914923014 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc144cc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 -- 172.21.15.29:0/2914923014 >> 172.21.15.29:0/2914923014 conn(0x7f56fc065ed0 msgr2=0x7f56fc0b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 -- 172.21.15.29:0/2914923014 shutdown_connections 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.242+0000 7f570168c640 1 -- 172.21.15.29:0/2914923014 wait complete. 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.243+0000 7f570168c640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.243+0000 7f570168c640 10 monclient: monmap: 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.243+0000 7f570168c640 1 librados: starting msgr at 2023-08-22T23:22:35.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.243+0000 7f570168c640 1 librados: starting objecter 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 1 Processor -- start 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 1 -- start start 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 1 librados: setting wanted keys 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 1 librados: calling monclient init 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 10 monclient: init 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:22:35.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56fc0add90 0x7f56fc13a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 10 monclient(hunting): picked mon.noname-b con 0x7f56fc0add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.244+0000 7f570168c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56fc13e190 0x7f56fc13a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): picked mon.noname-a con 0x7f56fc13e190 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): picked mon.noname-c con 0x7f56fc1448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc13feb0 con 0x7f56fc13e190 2023-08-22T23:22:35.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc13b0a0 con 0x7f56fc0add90 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc139580 con 0x7f56fc1448e0 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f570168c640 10 monclient(hunting): authenticate will time out at 1519.495512s 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.245+0000 7f56fb7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc0b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:43604/0 (socket says 172.21.15.29:43604) 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 1 -- 172.21.15.29:0/1264251198 learned_addr learned my addr 172.21.15.29:0/1264251198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): get_auth_request con 0x7f56fc1448e0 auth_method 0 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.246+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.249+0000 7f56fb7fe640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2023-08-22T23:22:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 1 -- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56fc0add90 msgr2=0x7f56fc13a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56fc0add90 0x7f56fc13a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 1 -- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56fc13e190 msgr2=0x7f56fc13a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56fc13e190 0x7f56fc13a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 1 monclient: found mon.noname-c 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.250+0000 7f56fb7fe640 1 -- 172.21.15.29:0/1264251198 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0094d0 con 0x7f56fc1448e0 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.252+0000 7f56fb7fe640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc0b4c10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c780 tx=0x7f56f400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.252+0000 7f56e37fe640 10 client.?.objecter ms_handle_connect 0x7f56fc1448e0 2023-08-22T23:22:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.252+0000 7f56e37fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4003d80 con 0x7f56fc1448e0 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: dump: 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.260 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 5 monclient: authenticate success, global_id 4316 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f400ce90 con 0x7f56fc1448e0 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f40052b0 con 0x7f56fc1448e0 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 10 monclient: _renew_subs 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc13b5c0 con 0x7f56fc1448e0 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56e37fe640 10 monclient: dump: 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f56f97fa640 4 set_mon_vals no callback set 2023-08-22T23:22:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 10 monclient: _renew_subs 2023-08-22T23:22:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc0b5330 con 0x7f56fc1448e0 2023-08-22T23:22:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.256+0000 7f570168c640 1 librados: init done 2023-08-22T23:22:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.257+0000 7f570168c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.257+0000 7f570168c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.257+0000 7f570168c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.257+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56fc13fff0 con 0x7f56fc1448e0 2023-08-22T23:22:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.262+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc0b0007550 con 0x7fc0c013e8a0 2023-08-22T23:22:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.263+0000 7fc0b67fc640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fc0a406ba30 0x7fc0a406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.263+0000 7fc0c5a1f640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fc0a406ba30 0x7fc0a406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.263+0000 7fc0c5a1f640 10 monclient: get_auth_request con 0x7fc0a406ba30 auth_method 0 2023-08-22T23:22:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.263+0000 7fc0c5a1f640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fc0a406ba30 0x7fc0a406def0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc000f60 tx=0x7fc0bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.263+0000 7fc0b67fc640 10 client.4348.objecter ms_handle_connect 0x7fc0a406ba30 2023-08-22T23:22:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.266+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc0b005b2b0 con 0x7fc0c013e8a0 2023-08-22T23:22:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.266+0000 7fc0b67fc640 10 client.4348.objecter ms_dispatch 0x7fc0c0000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.266+0000 7fc0b67fc640 3 client.4348.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.266+0000 7fc0b67fc640 3 client.4348.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.266+0000 7fc0b67fc640 20 client.4348.objecter dump_active .. 0 homeless 2023-08-22T23:22:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.279+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fc0b00538e0 con 0x7fc0c013e8a0 2023-08-22T23:22:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.279+0000 7fc0b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.279+0000 7fc0b67fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:35.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-08-22T23:22:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.375+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff0bc03eb20 con 0x7ff0c41448e0 2023-08-22T23:22:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.376+0000 7ff0baffd640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff0a406ba30 0x7ff0a406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.376+0000 7ff0c259c640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff0a406ba30 0x7ff0a406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.376+0000 7ff0c259c640 10 monclient: get_auth_request con 0x7ff0a406ba30 auth_method 0 2023-08-22T23:22:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.377+0000 7ff0c259c640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff0a406ba30 0x7ff0a406def0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4006fd0 tx=0x7ff0b4009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.377+0000 7ff0baffd640 10 client.4338.objecter ms_handle_connect 0x7ff0a406ba30 2023-08-22T23:22:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.393+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff0bc07b960 con 0x7ff0c41448e0 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 10 client.4338.objecter ms_dispatch 0x7ff0c4000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 3 client.4338.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 3 client.4338.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 20 client.4338.objecter dump_active .. 0 homeless 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7ff0bc077610 con 0x7ff0c41448e0 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.394+0000 7ff0baffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:35.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.440+0000 7fabbdb99640 10 monclient: get_monmap_and_config 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.440+0000 7fabbdb99640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.440+0000 7fabbdb99640 10 monclient: monmap: 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:35.441515+0000 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:35.441515+0000 2023-08-22T23:22:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:35.443 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 1 Processor -- start 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 1 -- start start 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient: init 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabb81448e0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fabb80add90 0x7fabb813e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient(hunting): picked mon.noname-c con 0x7fabb80add90 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabb813e6d0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb80b12d0 con 0x7fabb813e6d0 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb80afae0 con 0x7fabb81448e0 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80afdc0 con 0x7fabb80add90 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.441+0000 7fabbdb99640 10 monclient(hunting): authenticate will time out at 1519.692228s 2023-08-22T23:22:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.442+0000 7fabb6ffd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fabb80add90 0x7fabb813e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.442+0000 7fabb77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fabb81448e0 0x7fabb8144cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44646/0 (socket says 172.21.15.29:44646) 2023-08-22T23:22:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.442+0000 7fabb77fe640 1 -- 172.21.15.29:0/279000133 learned_addr learned my addr 172.21.15.29:0/279000133 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): get_auth_request con 0x7fabb80add90 auth_method 0 2023-08-22T23:22:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.443+0000 7fabb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2023-08-22T23:22:35.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 1 -- 172.21.15.29:0/279000133 >> v1:172.21.15.52:6789/0 conn(0x7fabb81448e0 legacy=0x7fabb8144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 1 -- 172.21.15.29:0/279000133 >> v1:172.21.15.29:6789/0 conn(0x7fabb813e6d0 legacy=0x7fabb814ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 1 monclient: found mon.noname-c 2023-08-22T23:22:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.445+0000 7fabb6ffd640 1 -- 172.21.15.29:0/279000133 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb80b00a0 con 0x7fabb80add90 2023-08-22T23:22:35.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-08-22T23:22:35.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.448+0000 7fabb6ffd640 1 --2- 172.21.15.29:0/279000133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb813e190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabac0097a0 tx=0x7fabac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6178d8cc03e0bb32 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-08-22T23:22:35.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 1 -- 172.21.15.29:0/279000133 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac031830 con 0x7fabb80add90 2023-08-22T23:22:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:22:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: dump: 2023-08-22T23:22:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 1 -- 172.21.15.29:0/279000133 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac031990 con 0x7fabb80add90 2023-08-22T23:22:35.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 1 -- 172.21.15.29:0/279000133 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac031d20 con 0x7fabb80add90 2023-08-22T23:22:35.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:35.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabb67fc640 10 monclient: dump: 2023-08-22T23:22:35.458 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.459 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.459 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.459 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.459 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.459 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.459 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.460 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.460 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.460 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabbdb99640 5 monclient: authenticate success, global_id 4351 2023-08-22T23:22:35.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabbdb99640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:35.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.453+0000 7fabbdb99640 4 set_mon_vals no callback set 2023-08-22T23:22:35.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 10 monclient: shutdown 2023-08-22T23:22:35.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 -- 172.21.15.29:0/279000133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 msgr2=0x7fabb813e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 --2- 172.21.15.29:0/279000133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb813e190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabac0097a0 tx=0x7fabac02b8d0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 -- 172.21.15.29:0/279000133 shutdown_connections 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 --2- 172.21.15.29:0/279000133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb813e190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 -- 172.21.15.29:0/279000133 >> 172.21.15.29:0/279000133 conn(0x7fabb8065ed0 msgr2=0x7fabb80b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 -- 172.21.15.29:0/279000133 shutdown_connections 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.454+0000 7fabbdb99640 1 -- 172.21.15.29:0/279000133 wait complete. 2023-08-22T23:22:35.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.455+0000 7fabbdb99640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.455+0000 7fabbdb99640 10 monclient: monmap: 2023-08-22T23:22:35.464 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.464 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:35.465 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:35.465 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:35.465 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.465 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.466 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:35.466 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:35.466 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:35.466 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.455+0000 7fabbdb99640 1 librados: starting msgr at 2023-08-22T23:22:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.455+0000 7fabbdb99640 1 librados: starting objecter 2023-08-22T23:22:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.456+0000 7fabbdb99640 1 Processor -- start 2023-08-22T23:22:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.456+0000 7fabbdb99640 1 -- start start 2023-08-22T23:22:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.456+0000 7fabbdb99640 1 librados: setting wanted keys 2023-08-22T23:22:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.456+0000 7fabbdb99640 1 librados: calling monclient init 2023-08-22T23:22:35.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.456+0000 7fabbdb99640 10 monclient: init 2023-08-22T23:22:35.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:22:35.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb8140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): picked mon.noname-b con 0x7fabb80add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:35.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fabb813e6d0 0x7fabb81434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): picked mon.noname-c con 0x7fabb813e6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb81448e0 0x7fabb8141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): picked mon.noname-a con 0x7fabb81448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8140f40 con 0x7fabb81448e0 2023-08-22T23:22:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb806d430 con 0x7fabb80add90 2023-08-22T23:22:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8143a00 con 0x7fabb813e6d0 2023-08-22T23:22:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.457+0000 7fabbdb99640 10 monclient(hunting): authenticate will time out at 1519.707848s 2023-08-22T23:22:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb8140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb8140770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39804/0 (socket says 172.21.15.29:39804) 2023-08-22T23:22:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 1 -- 172.21.15.29:0/1786424869 learned_addr learned my addr 172.21.15.29:0/1786424869 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): get_auth_request con 0x7fabb80add90 auth_method 0 2023-08-22T23:22:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb7fff640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb81448e0 0x7fabb8141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb7fff640 10 monclient(hunting): get_auth_request con 0x7fabb81448e0 auth_method 0 2023-08-22T23:22:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb7fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.458+0000 7fabb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.459+0000 7fabb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.459+0000 7fabb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.459+0000 7fabb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2023-08-22T23:22:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 1 -- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fabb813e6d0 msgr2=0x7fabb81434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fabb813e6d0 0x7fabb81434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 1 -- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb81448e0 msgr2=0x7fabb8141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb81448e0 0x7fabb8141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 1 monclient: found mon.noname-b 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.461+0000 7fabb77fe640 1 -- 172.21.15.29:0/1786424869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac009400 con 0x7fabb80add90 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.464+0000 7fabb77fe640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb8140770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faba800c480 tx=0x7faba800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.464+0000 7fab9ffff640 10 client.?.objecter ms_handle_connect 0x7fabb80add90 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.464+0000 7fab9ffff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8005ce0 con 0x7fabb80add90 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 10 monclient: dump: 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.481 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8005e40 con 0x7fabb80add90 2023-08-22T23:22:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 5 monclient: authenticate success, global_id 4354 2023-08-22T23:22:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 10 monclient: _renew_subs 2023-08-22T23:22:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb8144190 con 0x7fabb80add90 2023-08-22T23:22:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 10 monclient: _renew_subs 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb813f0d0 con 0x7fabb80add90 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.467+0000 7fabbdb99640 1 librados: init done 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.468+0000 7fab9ffff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.468+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba80072f0 con 0x7fabb80add90 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.468+0000 7fab9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.468+0000 7fab9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.468+0000 7fab9ffff640 10 monclient: dump: 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.468+0000 7fabb5ffb640 4 set_mon_vals no callback set 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.469+0000 7fabbdb99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.469+0000 7fabbdb99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.469+0000 7fabbdb99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.469+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab98005040 con 0x7fabb80add90 2023-08-22T23:22:35.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.490+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7faba8007510 con 0x7fabb80add90 2023-08-22T23:22:35.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.491+0000 7fab9ffff640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7faba406ba30 0x7faba406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.491+0000 7fabb6ffd640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7faba406ba30 0x7faba406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.491+0000 7fabb6ffd640 10 monclient: get_auth_request con 0x7faba406ba30 auth_method 0 2023-08-22T23:22:35.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fabb6ffd640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7faba406ba30 0x7faba406def0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fabac036000 tx=0x7fabac030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fab9ffff640 10 client.4354.objecter ms_handle_connect 0x7faba406ba30 2023-08-22T23:22:35.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faba805b110 con 0x7fabb80add90 2023-08-22T23:22:35.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fab9ffff640 10 client.4354.objecter ms_dispatch 0x7fabb8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:35.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fab9ffff640 3 client.4354.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:35.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fab9ffff640 3 client.4354.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:35.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.492+0000 7fab9ffff640 20 client.4354.objecter dump_active .. 0 homeless 2023-08-22T23:22:35.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.496+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f56f4005410 con 0x7f56fc1448e0 2023-08-22T23:22:35.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.496+0000 7f56e37fe640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56e806ba30 0x7f56e806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.496+0000 7f56faffd640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56e806ba30 0x7f56e806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.496+0000 7f56faffd640 10 monclient: get_auth_request con 0x7f56e806ba30 auth_method 0 2023-08-22T23:22:35.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.497+0000 7f56e37fe640 10 client.4316.objecter ms_handle_connect 0x7f56e806ba30 2023-08-22T23:22:35.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.497+0000 7f56faffd640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56e806ba30 0x7f56e806def0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56ec036000 tx=0x7f56ec030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.545+0000 7fc0c7c83640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-08-22T23:22:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.545+0000 7fc0c7c83640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-08-22T23:22:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.545+0000 7fc0c7c83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.545+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc0c01da000 con 0x7fc0c013e8a0 2023-08-22T23:22:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.562+0000 7f454c070640 10 monclient: get_monmap_and_config 2023-08-22T23:22:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.562+0000 7f454c070640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.562+0000 7f454c070640 10 monclient: monmap: 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:35.563494+0000 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:35.563494+0000 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:35.565 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.562+0000 7f454c070640 1 Processor -- start 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 1 -- start start 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 10 monclient: init 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f45441448e0 0x7f4544144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 10 monclient(hunting): picked mon.noname-a con 0x7f45441448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45440add90 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f454413e8a0 0x7f4544145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.563+0000 7f454c070640 10 monclient(hunting): picked mon.noname-b con 0x7f454413e8a0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45440b12d0 con 0x7f45440add90 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45440afae0 con 0x7f45441448e0 2023-08-22T23:22:35.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45440afdc0 con 0x7f454413e8a0 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f454c070640 10 monclient(hunting): authenticate will time out at 1519.814512s 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.564+0000 7f4549e0c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f45441448e0 0x7f4544144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.565+0000 7f454960b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f45440add90 0x7f454413e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60006/0 (socket says 172.21.15.29:60006) 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.565+0000 7f454960b640 1 -- 172.21.15.29:0/138945680 learned_addr learned my addr 172.21.15.29:0/138945680 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.565+0000 7f4549e0c640 10 monclient(hunting): get_auth_request con 0x7f45441448e0 auth_method 0 2023-08-22T23:22:35.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.565+0000 7f4549e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.565+0000 7f4549e0c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.565+0000 7f4549e0c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.566+0000 7f4549e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.566+0000 7f4549e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.566+0000 7f4549e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 1 -- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f454413e8a0 msgr2=0x7f4544145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 1 --2- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f454413e8a0 0x7f4544145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 1 -- 172.21.15.29:0/138945680 >> v1:172.21.15.29:6789/0 conn(0x7f45440add90 legacy=0x7f454413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 1 monclient: found mon.noname-a 2023-08-22T23:22:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.568+0000 7f4549e0c640 1 -- 172.21.15.29:0/138945680 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440b00a0 con 0x7f45441448e0 2023-08-22T23:22:35.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-08-22T23:22:35.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.570+0000 7f6378986640 10 monclient: get_monmap_and_config 2023-08-22T23:22:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.570+0000 7f6378986640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.570+0000 7f6378986640 10 monclient: monmap: 2023-08-22T23:22:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:35.571404+0000 2023-08-22T23:22:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:35.571404+0000 2023-08-22T23:22:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.570+0000 7f6378986640 1 Processor -- start 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 -- start start 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient: init 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63741448e0 0x7f6374144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): picked mon.noname-a con 0x7f63741448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f63740add90 0x7f6374145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): picked mon.noname-c con 0x7f63740add90 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f637413e190 0x7f637413e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): picked mon.noname-b con 0x7f637413e190 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63740b12d0 con 0x7f63741448e0 2023-08-22T23:22:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63740afae0 con 0x7f63740add90 2023-08-22T23:22:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63740afdc0 con 0x7f637413e190 2023-08-22T23:22:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.571+0000 7f6378986640 10 monclient(hunting): authenticate will time out at 1519.822332s 2023-08-22T23:22:35.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.573+0000 7f4549e0c640 1 --2- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45441448e0 0x7f4544144cc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45400097a0 tx=0x7f454002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f159606d00317035 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.574+0000 7f637259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63741448e0 0x7f6374144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.576+0000 7f637259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63741448e0 0x7f6374144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33616/0 (socket says 172.21.15.29:33616) 2023-08-22T23:22:35.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.576+0000 7f637259c640 1 -- 172.21.15.29:0/1356214505 learned_addr learned my addr 172.21.15.29:0/1356214505 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.576+0000 7f637259c640 10 monclient(hunting): get_auth_request con 0x7f63741448e0 auth_method 0 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.576+0000 7f637259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.576+0000 7f637259c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.576+0000 7f637259c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.577+0000 7f637259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.577+0000 7f637259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.577+0000 7f637259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2023-08-22T23:22:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 1 -- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f637413e190 msgr2=0x7f637413e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 1 --2- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f637413e190 0x7f637413e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 1 -- 172.21.15.29:0/1356214505 >> v2:172.21.15.52:3300/0 conn(0x7f63740add90 msgr2=0x7f6374145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 1 --2- 172.21.15.29:0/1356214505 >> v2:172.21.15.52:3300/0 conn(0x7f63740add90 0x7f6374145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 1 monclient: found mon.noname-a 2023-08-22T23:22:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.579+0000 7f637259c640 1 -- 172.21.15.29:0/1356214505 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63740b00a0 con 0x7f63741448e0 2023-08-22T23:22:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 1 -- 172.21.15.29:0/138945680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454002d040 con 0x7f45441448e0 2023-08-22T23:22:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: dump: 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 1 -- 172.21.15.29:0/138945680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4540031810 con 0x7f45441448e0 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 1 -- 172.21.15.29:0/138945680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540031ba0 con 0x7f45441448e0 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f4548e0a640 10 monclient: dump: 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f454c070640 5 monclient: authenticate success, global_id 4347 2023-08-22T23:22:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f454c070640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.581+0000 7f454c070640 4 set_mon_vals no callback set 2023-08-22T23:22:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 10 monclient: shutdown 2023-08-22T23:22:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 -- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45441448e0 msgr2=0x7f4544144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 --2- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45441448e0 0x7f4544144cc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45400097a0 tx=0x7f454002b8b0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 -- 172.21.15.29:0/138945680 shutdown_connections 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 --2- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f454413e8a0 0x7f4544145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 --2- 172.21.15.29:0/138945680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45441448e0 0x7f4544144cc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 -- 172.21.15.29:0/138945680 >> 172.21.15.29:0/138945680 conn(0x7f4544065ed0 msgr2=0x7f45440b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 -- 172.21.15.29:0/138945680 shutdown_connections 2023-08-22T23:22:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.582+0000 7f454c070640 1 -- 172.21.15.29:0/138945680 wait complete. 2023-08-22T23:22:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.583+0000 7f637259c640 1 --2- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f6374144cc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f637413eec0 tx=0x7f636402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e86c900cbd1fe41 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.583+0000 7f454c070640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.583+0000 7f454c070640 10 monclient: monmap: 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 1 librados: starting msgr at 2023-08-22T23:22:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 1 librados: starting objecter 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 1 Processor -- start 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 1 -- start start 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 1 librados: setting wanted keys 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 1 librados: calling monclient init 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.584+0000 7f454c070640 10 monclient: init 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45440add90 0x7f454413d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): picked mon.noname-c con 0x7f45440add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 0x7f4544130a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): picked mon.noname-a con 0x7f454413e8a0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45441448e0 0x7f4544130fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): picked mon.noname-b con 0x7f45441448e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544143f70 con 0x7f454413e8a0 2023-08-22T23:22:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544144250 con 0x7f45441448e0 2023-08-22T23:22:35.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441314f0 con 0x7f45440add90 2023-08-22T23:22:35.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.585+0000 7f454c070640 10 monclient(hunting): authenticate will time out at 1519.835902s 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 1 -- 172.21.15.29:0/1356214505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364031870 con 0x7f63741448e0 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: dump: 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.596 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.597 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 1 -- 172.21.15.29:0/1356214505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63640319d0 con 0x7f63741448e0 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 1 -- 172.21.15.29:0/1356214505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364031d60 con 0x7f63741448e0 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6371d9b640 10 monclient: dump: 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6378986640 5 monclient: authenticate success, global_id 4353 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6378986640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f6378986640 4 set_mon_vals no callback set 2023-08-22T23:22:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.589+0000 7f454960b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 0x7f4544130a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 10 monclient: shutdown 2023-08-22T23:22:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 -- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 msgr2=0x7f6374144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 --2- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f6374144cc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f637413eec0 tx=0x7f636402b910 comp rx=0 tx=0).stop 2023-08-22T23:22:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 -- 172.21.15.29:0/1356214505 shutdown_connections 2023-08-22T23:22:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 --2- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f637413e190 0x7f637413e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 --2- 172.21.15.29:0/1356214505 >> v2:172.21.15.52:3300/0 conn(0x7f63740add90 0x7f6374145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 --2- 172.21.15.29:0/1356214505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f6374144cc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 -- 172.21.15.29:0/1356214505 >> 172.21.15.29:0/1356214505 conn(0x7f6374065ed0 msgr2=0x7f63740b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f454960b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 0x7f4544130a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33622/0 (socket says 172.21.15.29:33622) 2023-08-22T23:22:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f454960b640 1 -- 172.21.15.29:0/1606220627 learned_addr learned my addr 172.21.15.29:0/1606220627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f454960b640 10 monclient(hunting): get_auth_request con 0x7f454413e8a0 auth_method 0 2023-08-22T23:22:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f454960b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f454960b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f454960b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 -- 172.21.15.29:0/1356214505 shutdown_connections 2023-08-22T23:22:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.590+0000 7f6378986640 1 -- 172.21.15.29:0/1356214505 wait complete. 2023-08-22T23:22:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 10 monclient: monmap: 2023-08-22T23:22:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:35.605 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f454960b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f454960b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f454960b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 1 librados: starting msgr at 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 1 librados: starting objecter 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 1 Processor -- start 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 1 -- start start 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 1 librados: setting wanted keys 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 1 librados: calling monclient init 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.591+0000 7f6378986640 10 monclient: init 2023-08-22T23:22:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f63740add90 0x7f6374143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): picked mon.noname-c con 0x7f63740add90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f637413e190 0x7f63741daa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): picked mon.noname-b con 0x7f637413e190 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f63741daf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): picked mon.noname-a con 0x7f63741448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63741dd8f0 con 0x7f63741448e0 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63740b4750 con 0x7f637413e190 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6374142cf0 con 0x7f63740add90 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.592+0000 7f6378986640 10 monclient(hunting): authenticate will time out at 1519.842968s 2023-08-22T23:22:35.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2023-08-22T23:22:35.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 1 -- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45440add90 msgr2=0x7f454413d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45440add90 0x7f454413d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 1 -- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45441448e0 msgr2=0x7f4544130fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45441448e0 0x7f4544130fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 1 monclient: found mon.noname-a 2023-08-22T23:22:35.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.596+0000 7f454960b640 1 -- 172.21.15.29:0/1606220627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540009400 con 0x7f454413e8a0 2023-08-22T23:22:35.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.600+0000 7f6372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f63741daf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.600+0000 7f454960b640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 0x7f4544130a70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f453400c400 tx=0x7f453400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.600+0000 7f453a7fc640 10 client.?.objecter ms_handle_connect 0x7f454413e8a0 2023-08-22T23:22:35.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.600+0000 7f453a7fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.601+0000 7f6372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f63741daf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33626/0 (socket says 172.21.15.29:33626) 2023-08-22T23:22:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.601+0000 7f6372d9d640 1 -- 172.21.15.29:0/1512781893 learned_addr learned my addr 172.21.15.29:0/1512781893 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.601+0000 7f6372d9d640 10 monclient(hunting): get_auth_request con 0x7f63741448e0 auth_method 0 2023-08-22T23:22:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.601+0000 7f6372d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.601+0000 7f6372d9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.601+0000 7f6372d9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.602+0000 7f6372d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.602+0000 7f6372d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.602+0000 7f6372d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.606+0000 7f6372d9d640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.606+0000 7f6372d9d640 1 -- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f63740add90 msgr2=0x7f6374143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.606+0000 7f6372d9d640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f63740add90 0x7f6374143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.606+0000 7f6372d9d640 1 -- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f637413e190 msgr2=0x7f63741daa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.606+0000 7f6372d9d640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f637413e190 0x7f63741daa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.607+0000 7f6372d9d640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.607+0000 7f6372d9d640 1 monclient: found mon.noname-a 2023-08-22T23:22:35.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.607+0000 7f6372d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.607+0000 7f6372d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.607+0000 7f6372d9d640 1 -- 172.21.15.29:0/1512781893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640094d0 con 0x7f63741448e0 2023-08-22T23:22:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.609+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534005ce0 con 0x7f454413e8a0 2023-08-22T23:22:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.609+0000 7f453a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.609+0000 7f453a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.609+0000 7f453a7fc640 10 monclient: dump: 2023-08-22T23:22:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.609+0000 7f453a7fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.609+0000 7f453a7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4534005e40 con 0x7f454413e8a0 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 5 monclient: authenticate success, global_id 4356 2023-08-22T23:22:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340072d0 con 0x7f454413e8a0 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453a7fc640 10 monclient: dump: 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.619 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f453bfff640 4 set_mon_vals no callback set 2023-08-22T23:22:35.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 10 monclient: _renew_subs 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45441428b0 con 0x7f454413e8a0 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 10 monclient: _renew_subs 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4544143d90 con 0x7f454413e8a0 2023-08-22T23:22:35.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.610+0000 7f454c070640 1 librados: init done 2023-08-22T23:22:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.611+0000 7f454c070640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.611+0000 7f454c070640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.611+0000 7f454c070640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.611+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4544066cb0 con 0x7f454413e8a0 2023-08-22T23:22:35.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.687+0000 7fa872b5d640 10 monclient: get_monmap_and_config 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.687+0000 7fa872b5d640 10 monclient: build_initial_monmap 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.687+0000 7fa872b5d640 10 monclient: monmap: 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:35.688531+0000 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:35.688531+0000 2023-08-22T23:22:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:35.690 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.687+0000 7fa872b5d640 1 Processor -- start 2023-08-22T23:22:35.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 1 -- start start 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient: init 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa86c0a7d50 0x7fa86c0a8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa86c0a7d50 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa86c0aff90 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa86c0b03e0 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c13ee50 con 0x7fa86c0b03e0 2023-08-22T23:22:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c0a9df0 con 0x7fa86c0aff90 2023-08-22T23:22:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c0a87c0 con 0x7fa86c0a7d50 2023-08-22T23:22:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.688+0000 7fa872b5d640 10 monclient(hunting): authenticate will time out at 1519.939207s 2023-08-22T23:22:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.703+0000 7ff0c8fc4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-08-22T23:22:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.703+0000 7ff0c8fc4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-08-22T23:22:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.703+0000 7ff0c8fc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.703+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff0c40af770 con 0x7ff0c41448e0 2023-08-22T23:22:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.714+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f56f4057bb0 con 0x7f56fc1448e0 2023-08-22T23:22:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.714+0000 7f56e37fe640 10 client.4316.objecter ms_dispatch 0x7f56fc000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.714+0000 7f56e37fe640 3 client.4316.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.714+0000 7f56e37fe640 3 client.4316.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.715+0000 7f56e37fe640 20 client.4316.objecter dump_active .. 0 homeless 2023-08-22T23:22:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.715+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f56f4053860 con 0x7f56fc1448e0 2023-08-22T23:22:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.715+0000 7f56e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.715+0000 7f56e37fe640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:35.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.747+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7faba8053740 con 0x7fabb80add90 2023-08-22T23:22:35.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.747+0000 7fab9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.747+0000 7fab9ffff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:35.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.751+0000 7fa86bfff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fa86c0aff90 0x7fa86c1411e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44662/0 (socket says 172.21.15.29:44662) 2023-08-22T23:22:35.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.751+0000 7fa86bfff640 1 -- 172.21.15.29:0/3839693709 learned_addr learned my addr 172.21.15.29:0/3839693709 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:35.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.753+0000 7fc0b67fc640 1 -- 172.21.15.29:0/2003751952 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0c01da000 con 0x7fc0c013e8a0 2023-08-22T23:22:35.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.753+0000 7fc0b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-08-22T23:22:35.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.753+0000 7fc0b67fc640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:35.756 INFO:teuthology.orchestra.run.smithi029.stdout:64424509446 2023-08-22T23:22:35.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 10 librados: watch_flush enter 2023-08-22T23:22:35.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 10 librados: watch_flush exit 2023-08-22T23:22:35.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 20 client.4348.objecter shutdown clearing up homeless session... 2023-08-22T23:22:35.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 10 client.4348.objecter successfully canceled tick 2023-08-22T23:22:35.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fc0a406ba30 msgr2=0x7fc0a406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fc0a406ba30 0x7fc0a406def0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc000f60 tx=0x7fc0bc009040 comp rx=0 tx=0).stop 2023-08-22T23:22:35.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 10 monclient: shutdown 2023-08-22T23:22:35.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 msgr2=0x7fc0c01d9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c01d9560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c500 tx=0x7fc0b000c9c0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 shutdown_connections 2023-08-22T23:22:35.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fc0a406ba30 0x7fc0a406def0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c01448e0 0x7fc0c01d5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0c013e8a0 0x7fc0c01d9560 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 --2- 172.21.15.29:0/2003751952 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fc0c00add90 0x7fc0c0143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 >> 172.21.15.29:0/2003751952 conn(0x7fc0c0065ed0 msgr2=0x7fc0c00b26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 shutdown_connections 2023-08-22T23:22:35.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 -- 172.21.15.29:0/2003751952 wait complete. 2023-08-22T23:22:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.755+0000 7fc0c7c83640 1 librados: shutdown 2023-08-22T23:22:35.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.4 2023-08-22T23:22:35.768 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:35.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.878+0000 7f6372d9d640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f63741daf60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f636c00b970 tx=0x7f636c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.878+0000 7f636affd640 10 client.?.objecter ms_handle_connect 0x7f63741448e0 2023-08-22T23:22:35.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.878+0000 7f636affd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:35.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.893+0000 7f570168c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-08-22T23:22:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.893+0000 7f570168c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-08-22T23:22:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.893+0000 7f570168c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:22:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.893+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f56fc13fff0 con 0x7f56fc1448e0 2023-08-22T23:22:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.893+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4534007430 con 0x7f454413e8a0 2023-08-22T23:22:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.894+0000 7f453a7fc640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f452806ba10 0x7f452806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.894+0000 7f4549e0c640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f452806ba10 0x7f452806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.894+0000 7f4549e0c640 10 monclient: get_auth_request con 0x7f452806ba10 auth_method 0 2023-08-22T23:22:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.895+0000 7f4549e0c640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f452806ba10 0x7f452806ded0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4540036000 tx=0x7f4540030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.895+0000 7f453a7fc640 10 client.4356.objecter ms_handle_connect 0x7f452806ba10 2023-08-22T23:22:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.897+0000 7f56e37fe640 1 -- 172.21.15.29:0/1264251198 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f56fc13fff0 con 0x7f56fc1448e0 2023-08-22T23:22:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.897+0000 7f56e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-08-22T23:22:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.897+0000 7f56e37fe640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:35.899 INFO:teuthology.orchestra.run.smithi029.stdout:60129542150 2023-08-22T23:22:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 10 librados: watch_flush enter 2023-08-22T23:22:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 10 librados: watch_flush exit 2023-08-22T23:22:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 20 client.4316.objecter shutdown clearing up homeless session... 2023-08-22T23:22:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 10 client.4316.objecter successfully canceled tick 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56e806ba30 msgr2=0x7f56e806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56e806ba30 0x7f56e806def0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56ec036000 tx=0x7f56ec030000 comp rx=0 tx=0).stop 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 10 monclient: shutdown 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 msgr2=0x7f56fc0b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc0b4c10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c780 tx=0x7f56f400cc40 comp rx=0 tx=0).stop 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 shutdown_connections 2023-08-22T23:22:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f56e806ba30 0x7f56e806def0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f56fc1448e0 0x7f56fc0b4c10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56fc13e190 0x7f56fc13a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 --2- 172.21.15.29:0/1264251198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56fc0add90 0x7f56fc13a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 >> 172.21.15.29:0/1264251198 conn(0x7f56fc065ed0 msgr2=0x7f56fc144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 shutdown_connections 2023-08-22T23:22:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 -- 172.21.15.29:0/1264251198 wait complete. 2023-08-22T23:22:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.899+0000 7f570168c640 1 librados: shutdown 2023-08-22T23:22:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4534057a60 con 0x7f454413e8a0 2023-08-22T23:22:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 10 client.4356.objecter ms_dispatch 0x7f4544000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 3 client.4356.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 3 client.4356.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 20 client.4356.objecter dump_active .. 0 homeless 2023-08-22T23:22:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f4534053710 con 0x7f454413e8a0 2023-08-22T23:22:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.907+0000 7f453a7fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:35.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2023-08-22T23:22:35.912 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.919+0000 7fa8708f9640 1 --2- 172.21.15.29:0/3839693709 >> v2:172.21.15.29:3300/0 conn(0x7fa86c0a7d50 0x7fa86c0a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c00cb40 con 0x7f63741448e0 2023-08-22T23:22:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: dump: 2023-08-22T23:22:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: _finish_auth 0 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c00ccc0 con 0x7f63741448e0 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f6378986640 5 monclient: authenticate success, global_id 4359 2023-08-22T23:22:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c014680 con 0x7f63741448e0 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f636affd640 10 monclient: dump: 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.925 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.920+0000 7f637159a640 4 set_mon_vals no callback set 2023-08-22T23:22:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 10 monclient: _renew_subs 2023-08-22T23:22:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6374142470 con 0x7f63741448e0 2023-08-22T23:22:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 10 monclient: _renew_subs 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63741441b0 con 0x7f63741448e0 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.921+0000 7f6378986640 1 librados: init done 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.922+0000 7f6378986640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.922+0000 7f6378986640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.922+0000 7f6378986640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.922+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6374142630 con 0x7f63741448e0 2023-08-22T23:22:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.945+0000 7fabbdb99640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-08-22T23:22:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.945+0000 7fabbdb99640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-08-22T23:22:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.945+0000 7fabbdb99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.945+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fab980055c0 con 0x7fabb80add90 2023-08-22T23:22:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.949+0000 7fab9ffff640 1 -- 172.21.15.29:0/1786424869 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faba8017020 con 0x7fabb80add90 2023-08-22T23:22:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.949+0000 7fab9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-08-22T23:22:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.949+0000 7fab9ffff640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:35.951 INFO:teuthology.orchestra.run.smithi029.stdout:68719476742 2023-08-22T23:22:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 10 librados: watch_flush enter 2023-08-22T23:22:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 10 librados: watch_flush exit 2023-08-22T23:22:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 20 client.4354.objecter shutdown clearing up homeless session... 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 10 client.4354.objecter successfully canceled tick 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7faba406ba30 msgr2=0x7faba406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7faba406ba30 0x7faba406def0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fabac036000 tx=0x7fabac030000 comp rx=0 tx=0).stop 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 10 monclient: shutdown 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 msgr2=0x7fabb8140770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb8140770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faba800c480 tx=0x7faba800c940 comp rx=0 tx=0).stop 2023-08-22T23:22:35.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 shutdown_connections 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7faba406ba30 0x7faba406def0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb81448e0 0x7fabb8141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fabb813e6d0 0x7fabb81434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 --2- 172.21.15.29:0/1786424869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fabb80add90 0x7fabb8140770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 >> 172.21.15.29:0/1786424869 conn(0x7fabb8065ed0 msgr2=0x7fabb813a640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.951+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 shutdown_connections 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.952+0000 7fabbdb99640 1 -- 172.21.15.29:0/1786424869 wait complete. 2023-08-22T23:22:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.952+0000 7fabbdb99640 1 librados: shutdown 2023-08-22T23:22:35.963 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476742 for osd.0 2023-08-22T23:22:35.964 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.972+0000 7fa8708f9640 10 monclient(hunting): get_auth_request con 0x7fa86c0a7d50 auth_method 0 2023-08-22T23:22:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.972+0000 7fa8708f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.972+0000 7fa8708f9640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:35.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.972+0000 7fa8708f9640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.983+0000 7fa8708f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.983+0000 7fa8708f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.983+0000 7fa8708f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2023-08-22T23:22:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 1 -- 172.21.15.29:0/3839693709 >> v1:172.21.15.52:6789/0 conn(0x7fa86c0aff90 legacy=0x7fa86c1411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 1 -- 172.21.15.29:0/3839693709 >> v1:172.21.15.29:6789/0 conn(0x7fa86c0b03e0 legacy=0x7fa86c14ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 10 monclient: _finish_hunting 0 2023-08-22T23:22:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 1 monclient: found mon.noname-a 2023-08-22T23:22:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:35.987+0000 7fa8708f9640 1 -- 172.21.15.29:0/3839693709 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0a8d00 con 0x7fa86c0a7d50 2023-08-22T23:22:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.094+0000 7f454c070640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-08-22T23:22:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.094+0000 7f454c070640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-08-22T23:22:36.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.094+0000 7f454c070640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:36.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.094+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f45441443c0 con 0x7f454413e8a0 2023-08-22T23:22:36.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.242+0000 7fa8708f9640 1 --2- 172.21.15.29:0/3839693709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 0x7fa86c0a8130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa8600097a0 tx=0x7fa86002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3749d574c3cab2f server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 1 -- 172.21.15.29:0/3839693709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86002e070 con 0x7fa86c0a7d50 2023-08-22T23:22:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: dump: 2023-08-22T23:22:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.264 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: _finish_auth 0 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 1 -- 172.21.15.29:0/3839693709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8600315a0 con 0x7fa86c0a7d50 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa872b5d640 5 monclient: authenticate success, global_id 4362 2023-08-22T23:22:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa872b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 1 -- 172.21.15.29:0/3839693709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860031930 con 0x7fa86c0a7d50 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa86b7fe640 10 monclient: dump: 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.266 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa872b5d640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.261+0000 7fa872b5d640 4 set_mon_vals no callback set 2023-08-22T23:22:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7ff0baffd640 1 -- 172.21.15.29:0/1375339479 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff0bc043090 con 0x7ff0c41448e0 2023-08-22T23:22:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7ff0baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-08-22T23:22:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7ff0baffd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 10 monclient: shutdown 2023-08-22T23:22:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 -- 172.21.15.29:0/3839693709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 msgr2=0x7fa86c0a8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 --2- 172.21.15.29:0/3839693709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 0x7fa86c0a8130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa8600097a0 tx=0x7fa86002b840 comp rx=0 tx=0).stop 2023-08-22T23:22:36.270 INFO:teuthology.orchestra.run.smithi029.stdout:60129542149 2023-08-22T23:22:36.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 -- 172.21.15.29:0/3839693709 shutdown_connections 2023-08-22T23:22:36.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 --2- 172.21.15.29:0/3839693709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 0x7fa86c0a8130 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 -- 172.21.15.29:0/3839693709 >> 172.21.15.29:0/3839693709 conn(0x7fa86c065dd0 msgr2=0x7fa86c0b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 -- 172.21.15.29:0/3839693709 shutdown_connections 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.262+0000 7fa872b5d640 1 -- 172.21.15.29:0/3839693709 wait complete. 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 10 monclient: build_initial_monmap 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 10 monclient: monmap: 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:36.272 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 1 librados: starting msgr at 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 1 librados: starting objecter 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 1 Processor -- start 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 1 -- start start 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 1 librados: setting wanted keys 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 1 librados: calling monclient init 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.263+0000 7fa872b5d640 10 monclient: init 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 0x7fa86c0ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa86c0a7d50 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 0x7fa86c13efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa86c0aff90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa86c0b03e0 0x7fa86c0acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa86c0b03e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:36.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1c8f30 con 0x7fa86c0a7d50 2023-08-22T23:22:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1c8a10 con 0x7fa86c0aff90 2023-08-22T23:22:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 10 librados: watch_flush enter 2023-08-22T23:22:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1d9560 con 0x7fa86c0b03e0 2023-08-22T23:22:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7fa872b5d640 10 monclient(hunting): authenticate will time out at 1520.514830s 2023-08-22T23:22:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 10 librados: watch_flush exit 2023-08-22T23:22:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 20 client.4338.objecter shutdown clearing up homeless session... 2023-08-22T23:22:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 10 client.4338.objecter successfully canceled tick 2023-08-22T23:22:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff0a406ba30 msgr2=0x7ff0a406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff0a406ba30 0x7ff0a406def0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4006fd0 tx=0x7ff0b4009060 comp rx=0 tx=0).stop 2023-08-22T23:22:36.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 10 monclient: shutdown 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 msgr2=0x7ff0c4141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 0x7ff0c4141810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc036040 tx=0x7ff0bc004a40 comp rx=0 tx=0).stop 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 shutdown_connections 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ff0a406ba30 0x7ff0a406def0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0c41448e0 0x7ff0c4141810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff0c413e6d0 0x7ff0c41412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 --2- 172.21.15.29:0/1375339479 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ff0c40add90 0x7ff0c4144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 >> 172.21.15.29:0/1375339479 conn(0x7ff0c4065ed0 msgr2=0x7ff0c413a030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 shutdown_connections 2023-08-22T23:22:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 -- 172.21.15.29:0/1375339479 wait complete. 2023-08-22T23:22:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.264+0000 7ff0c8fc4640 1 librados: shutdown 2023-08-22T23:22:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 0x7fa86c13efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa8710fa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa86c0b03e0 0x7fa86c0acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 0x7fa86c13efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39840/0 (socket says 172.21.15.29:39840) 2023-08-22T23:22:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 1 -- 172.21.15.29:0/2307137414 learned_addr learned my addr 172.21.15.29:0/2307137414 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): get_auth_request con 0x7fa86c0aff90 auth_method 0 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa8710fa640 10 monclient(hunting): get_auth_request con 0x7fa86c0b03e0 auth_method 0 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa8710fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa8710fa640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa8710fa640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.265+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.266+0000 7fa8710fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.266+0000 7fa8710fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.266+0000 7fa8710fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.268+0000 7f453a7fc640 1 -- 172.21.15.29:0/1606220627 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4534017070 con 0x7f454413e8a0 2023-08-22T23:22:36.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.268+0000 7f453a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-08-22T23:22:36.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.268+0000 7f453a7fc640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:36.284 INFO:teuthology.orchestra.run.smithi029.stdout:68719476742 2023-08-22T23:22:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.268+0000 7fa86bfff640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-08-22T23:22:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 1 -- 172.21.15.29:0/2307137414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa86c0b03e0 msgr2=0x7fa86c0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa86c0b03e0 0x7fa86c0acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 1 -- 172.21.15.29:0/2307137414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 msgr2=0x7fa86c0ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 0x7fa86c0ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 1 monclient: found mon.noname-b 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.269+0000 7fa86bfff640 1 -- 172.21.15.29:0/2307137414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860009400 con 0x7fa86c0aff90 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7fa86bfff640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 0x7fa86c13efe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c4a0 tx=0x7fa85800c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7fa8697fa640 10 client.?.objecter ms_handle_connect 0x7fa86c0aff90 2023-08-22T23:22:36.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7fa8697fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:36.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 10 librados: watch_flush enter 2023-08-22T23:22:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 10 librados: watch_flush exit 2023-08-22T23:22:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 20 client.4356.objecter shutdown clearing up homeless session... 2023-08-22T23:22:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 10 client.4356.objecter successfully canceled tick 2023-08-22T23:22:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f452806ba10 msgr2=0x7f452806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f452806ba10 0x7f452806ded0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4540036000 tx=0x7f4540030000 comp rx=0 tx=0).stop 2023-08-22T23:22:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 10 monclient: shutdown 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 msgr2=0x7f4544130a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.271+0000 7f454c070640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 0x7f4544130a70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f453400c400 tx=0x7f453400c8c0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 shutdown_connections 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f452806ba10 0x7f452806ded0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45441448e0 0x7f4544130fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f454413e8a0 0x7f4544130a70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 --2- 172.21.15.29:0/1606220627 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f45440add90 0x7f454413d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 >> 172.21.15.29:0/1606220627 conn(0x7f4544065ed0 msgr2=0x7f4544144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 shutdown_connections 2023-08-22T23:22:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 -- 172.21.15.29:0/1606220627 wait complete. 2023-08-22T23:22:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.272+0000 7f454c070640 1 librados: shutdown 2023-08-22T23:22:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa858005ce0 con 0x7fa86c0aff90 2023-08-22T23:22:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: dump: 2023-08-22T23:22:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: _finish_auth 0 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: _check_auth_tickets 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa858005e40 con 0x7fa86c0aff90 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 5 monclient: authenticate success, global_id 4357 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 10 monclient: _renew_subs 2023-08-22T23:22:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c0ad000 con 0x7fa86c0aff90 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa858007270 con 0x7fa86c0aff90 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa8697fa640 10 monclient: dump: 2023-08-22T23:22:36.294 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 10 monclient: _renew_subs 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa86affd640 4 set_mon_vals no callback set 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c142260 con 0x7fa86c0aff90 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.275+0000 7fa872b5d640 1 librados: init done 2023-08-22T23:22:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.276+0000 7fa872b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.276+0000 7fa872b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.276+0000 7fa872b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.276+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c0ad330 con 0x7fa86c0aff90 2023-08-22T23:22:36.298 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.5 2023-08-22T23:22:36.298 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:36.299 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476742 for osd.2 2023-08-22T23:22:36.299 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.304+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f636c01d3b0 con 0x7f63741448e0 2023-08-22T23:22:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.304+0000 7f636affd640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f635406b9e0 0x7f635406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.304+0000 7f637259c640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f635406b9e0 0x7f635406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.304+0000 7f637259c640 10 monclient: get_auth_request con 0x7f635406b9e0 auth_method 0 2023-08-22T23:22:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.305+0000 7f637259c640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f635406b9e0 0x7f635406dea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6364036000 tx=0x7f6364030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.305+0000 7f636affd640 10 client.4359.objecter ms_handle_connect 0x7f635406b9e0 2023-08-22T23:22:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f636c00ce30 con 0x7f63741448e0 2023-08-22T23:22:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 10 client.4359.objecter ms_dispatch 0x7f6374000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 3 client.4359.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 3 client.4359.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 20 client.4359.objecter dump_active .. 0 homeless 2023-08-22T23:22:36.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f636c057b60 con 0x7f63741448e0 2023-08-22T23:22:36.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:36.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.318+0000 7f636affd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:36.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.538+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa858007490 con 0x7fa86c0aff90 2023-08-22T23:22:36.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7fa8697fa640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fa85006ba30 0x7fa85006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7fa8708f9640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fa85006ba30 0x7fa85006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7fa8708f9640 10 monclient: get_auth_request con 0x7fa85006ba30 auth_method 0 2023-08-22T23:22:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7f6378986640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-08-22T23:22:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7f6378986640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-08-22T23:22:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7f6378986640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:36.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f63741dda30 con 0x7f63741448e0 2023-08-22T23:22:36.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7fa8708f9640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fa85006ba30 0x7fa85006def0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa860036000 tx=0x7fa860030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.539+0000 7fa8697fa640 10 client.4357.objecter ms_handle_connect 0x7fa85006ba30 2023-08-22T23:22:36.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.540+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa85805aed0 con 0x7fa86c0aff90 2023-08-22T23:22:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.540+0000 7fa8697fa640 10 client.4357.objecter ms_dispatch 0x7fa86c000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.540+0000 7fa8697fa640 3 client.4357.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.540+0000 7fa8697fa640 3 client.4357.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:36.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.541+0000 7fa8697fa640 20 client.4357.objecter dump_active .. 0 homeless 2023-08-22T23:22:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.543+0000 7f636affd640 1 -- 172.21.15.29:0/1512781893 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f63741dda30 con 0x7f63741448e0 2023-08-22T23:22:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.543+0000 7f636affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-08-22T23:22:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.543+0000 7f636affd640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:36.544 INFO:teuthology.orchestra.run.smithi029.stdout:64424509447 2023-08-22T23:22:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 10 librados: watch_flush enter 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 10 librados: watch_flush exit 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 20 client.4359.objecter shutdown clearing up homeless session... 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 10 client.4359.objecter successfully canceled tick 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f635406b9e0 msgr2=0x7f635406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f635406b9e0 0x7f635406dea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6364036000 tx=0x7f6364030000 comp rx=0 tx=0).stop 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 10 monclient: shutdown 2023-08-22T23:22:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 msgr2=0x7f63741daf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f63741daf60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f636c00b970 tx=0x7f636c00be30 comp rx=0 tx=0).stop 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 shutdown_connections 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f635406b9e0 0x7f635406dea0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63741448e0 0x7f63741daf60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f637413e190 0x7f63741daa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 --2- 172.21.15.29:0/1512781893 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f63740add90 0x7f6374143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 >> 172.21.15.29:0/1512781893 conn(0x7f6374065ed0 msgr2=0x7f6374144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:36.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 shutdown_connections 2023-08-22T23:22:36.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 -- 172.21.15.29:0/1512781893 wait complete. 2023-08-22T23:22:36.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.545+0000 7f6378986640 1 librados: shutdown 2023-08-22T23:22:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.553+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fa858053500 con 0x7fa86c0aff90 2023-08-22T23:22:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.553+0000 7fa8697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.553+0000 7fa8697fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:36.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.6 2023-08-22T23:22:36.558 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:36.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.739+0000 7fa872b5d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-08-22T23:22:36.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.739+0000 7fa872b5d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-08-22T23:22:36.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.739+0000 7fa872b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.739+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa86c1c8870 con 0x7fa86c0aff90 2023-08-22T23:22:36.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.743+0000 7fa8697fa640 1 -- 172.21.15.29:0/2307137414 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa86c1c8870 con 0x7fa86c0aff90 2023-08-22T23:22:36.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.743+0000 7fa8697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-08-22T23:22:36.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.743+0000 7fa8697fa640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:36.745 INFO:teuthology.orchestra.run.smithi029.stdout:68719476742 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 10 librados: watch_flush enter 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 10 librados: watch_flush exit 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 20 client.4357.objecter shutdown clearing up homeless session... 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 10 client.4357.objecter successfully canceled tick 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fa85006ba30 msgr2=0x7fa85006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fa85006ba30 0x7fa85006def0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa860036000 tx=0x7fa860030000 comp rx=0 tx=0).stop 2023-08-22T23:22:36.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 10 monclient: shutdown 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 msgr2=0x7fa86c13efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 0x7fa86c13efe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c4a0 tx=0x7fa85800c960 comp rx=0 tx=0).stop 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 shutdown_connections 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fa85006ba30 0x7fa85006def0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fa86c0b03e0 0x7fa86c0acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa86c0aff90 0x7fa86c13efe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 --2- 172.21.15.29:0/2307137414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0a7d50 0x7fa86c0ac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 >> 172.21.15.29:0/2307137414 conn(0x7fa86c065dd0 msgr2=0x7fa86c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 shutdown_connections 2023-08-22T23:22:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 -- 172.21.15.29:0/2307137414 wait complete. 2023-08-22T23:22:36.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.745+0000 7fa872b5d640 1 librados: shutdown 2023-08-22T23:22:36.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476742 for osd.1 2023-08-22T23:22:36.757 DEBUG:teuthology.parallel:result is None 2023-08-22T23:22:36.757 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-08-22T23:22:36.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:22:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.860+0000 7fb2afd0f640 10 monclient: get_monmap_and_config 2023-08-22T23:22:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.860+0000 7fb2afd0f640 10 monclient: build_initial_monmap 2023-08-22T23:22:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.860+0000 7fb2afd0f640 10 monclient: monmap: 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:36.862200+0000 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:36.862200+0000 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:36.864 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:36.864 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:36.864 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:36.864 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.861+0000 7fb2afd0f640 1 Processor -- start 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.861+0000 7fb2afd0f640 1 -- start start 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.861+0000 7fb2afd0f640 10 monclient: init 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2a80a7b90 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2a80b1380 addr v1:172.21.15.29:6789/0 2023-08-22T23:22:36.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a80a7f70 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80a9b00 con 0x7fb2a80b1380 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80a9de0 con 0x7fb2a80a7b90 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80aa0c0 con 0x7fb2a80a7f70 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.862+0000 7fb2afd0f640 10 monclient(hunting): authenticate will time out at 1521.112924s 2023-08-22T23:22:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ad2aa640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2a80b1380 0x7fb2a81417e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60030/0 (socket says 172.21.15.29:60030) 2023-08-22T23:22:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ad2aa640 1 -- 172.21.15.29:0/3904825922 learned_addr learned my addr 172.21.15.29:0/3904825922 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ae2ac640 1 --2- 172.21.15.29:0/3904825922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ae2ac640 10 monclient(hunting): get_auth_request con 0x7fb2a80a7f70 auth_method 0 2023-08-22T23:22:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ae2ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ae2ac640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.863+0000 7fb2ae2ac640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.864+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:36.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.864+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:36.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.864+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:36.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380788132 0 0) 0x7fb2a80a9de0 con 0x7fb2a80a7b90 2023-08-22T23:22:36.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): _negotiate 2023-08-22T23:22:36.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): my global_id is 4360 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb294003520 con 0x7fb2a80a7b90 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377180231 0 0) 0x7fb2a80a9b00 con 0x7fb2a80b1380 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): _negotiate 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 10 monclient(hunting): my global_id is 4368 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a80a9de0 con 0x7fb2a80b1380 2023-08-22T23:22:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/3904825922 >> v1:172.21.15.52:6789/0 conn(0x7fb2a80a7b90 legacy=0x7fb2a813e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/3904825922 >> v1:172.21.15.29:6789/0 conn(0x7fb2a80b1380 legacy=0x7fb2a81417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 10 monclient: _finish_hunting 0 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 1 monclient: found mon.noname-b 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.865+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/3904825922 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a81412c0 con 0x7fb2a80a7f70 2023-08-22T23:22:36.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.867+0000 7fb2ae2ac640 1 --2- 172.21.15.29:0/3904825922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a814ca40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00097d0 tx=0x7fb2a002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5ef0dd72b44e7d server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0031890 con 0x7fb2a80a7f70 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: dump: 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: _finish_auth 0 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a00319f0 con 0x7fb2a80a7f70 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 1 -- 172.21.15.29:0/3904825922 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0031d80 con 0x7fb2a80a7f70 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2acaa9640 10 monclient: dump: 2023-08-22T23:22:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2afd0f640 5 monclient: authenticate success, global_id 4322 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2afd0f640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.871+0000 7fb2afd0f640 4 set_mon_vals no callback set 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 10 monclient: shutdown 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 -- 172.21.15.29:0/3904825922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 msgr2=0x7fb2a814ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/3904825922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a814ca40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00097d0 tx=0x7fb2a002b930 comp rx=0 tx=0).stop 2023-08-22T23:22:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 -- 172.21.15.29:0/3904825922 shutdown_connections 2023-08-22T23:22:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/3904825922 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a814ca40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 -- 172.21.15.29:0/3904825922 >> 172.21.15.29:0/3904825922 conn(0x7fb2a80b6a00 msgr2=0x7fb2a80b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 -- 172.21.15.29:0/3904825922 shutdown_connections 2023-08-22T23:22:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.872+0000 7fb2afd0f640 1 -- 172.21.15.29:0/3904825922 wait complete. 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 10 monclient: build_initial_monmap 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 10 monclient: monmap: 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 1 librados: starting msgr at 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 1 librados: starting objecter 2023-08-22T23:22:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 1 Processor -- start 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 1 -- start start 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 1 librados: setting wanted keys 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 1 librados: calling monclient init 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.873+0000 7fb2afd0f640 10 monclient: init 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:22:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a80a7b90 0x7fb2a813ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a80a7b90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a81d91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a80a7f70 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 0x7fb2a81d96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a80b1380 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a81447c0 con 0x7fb2a80a7b90 2023-08-22T23:22:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a813b710 con 0x7fb2a80b1380 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a81439a0 con 0x7fb2a80a7f70 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2afd0f640 10 monclient(hunting): authenticate will time out at 1521.124788s 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2adaab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a80a7b90 0x7fb2a813ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2ae2ac640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 0x7fb2a81d96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.874+0000 7fb2ad2aa640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a81d91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 0x7fb2a81d96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:39842/0 (socket says 172.21.15.29:39842) 2023-08-22T23:22:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/1233187875 learned_addr learned my addr 172.21.15.29:0/1233187875 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): get_auth_request con 0x7fb2a80b1380 auth_method 0 2023-08-22T23:22:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): get_auth_request con 0x7fb2a80a7b90 auth_method 0 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): get_auth_request con 0x7fb2a80a7f70 auth_method 0 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2adaab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.875+0000 7fb2ad2aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/1233187875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 msgr2=0x7fb2a81d91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a81d91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/1233187875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a80a7b90 msgr2=0x7fb2a813ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a80a7b90 0x7fb2a813ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 10 monclient: _finish_hunting 0 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 1 monclient: found mon.noname-b 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.878+0000 7fb2ae2ac640 1 -- 172.21.15.29:0/1233187875 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a0009400 con 0x7fb2a80b1380 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.880+0000 7fb2ae2ac640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 0x7fb2a81d96e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00097d0 tx=0x7fb2a00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.880+0000 7fb29e7fc640 10 client.?.objecter ms_handle_connect 0x7fb2a80b1380 2023-08-22T23:22:36.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.880+0000 7fb29e7fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:36.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a003bb70 con 0x7fb2a80b1380 2023-08-22T23:22:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: dump: 2023-08-22T23:22:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a003bcd0 con 0x7fb2a80b1380 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0045630 con 0x7fb2a80b1380 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 5 monclient: authenticate success, global_id 4363 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 10 monclient: _renew_subs 2023-08-22T23:22:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a81442a0 con 0x7fb2a80b1380 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29e7fc640 10 monclient: dump: 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb29ffff640 4 set_mon_vals no callback set 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 10 monclient: _renew_subs 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a8144ce0 con 0x7fb2a80b1380 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.884+0000 7fb2afd0f640 1 librados: init done 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.885+0000 7fb2afd0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.885+0000 7fb2afd0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.885+0000 7fb2afd0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.885+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a804dbd0 con 0x7fb2a80b1380 2023-08-22T23:22:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.905+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb2a004e450 con 0x7fb2a80b1380 2023-08-22T23:22:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.905+0000 7fb29e7fc640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb28c06ba30 0x7fb28c06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.905+0000 7fb2adaab640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb28c06ba30 0x7fb28c06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.905+0000 7fb2adaab640 10 monclient: get_auth_request con 0x7fb28c06ba30 auth_method 0 2023-08-22T23:22:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.906+0000 7fb29e7fc640 10 client.4363.objecter ms_handle_connect 0x7fb28c06ba30 2023-08-22T23:22:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.906+0000 7fb2adaab640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb28c06ba30 0x7fb28c06def0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4001560 tx=0x7fb2a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.906+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb2a007e140 con 0x7fb2a80b1380 2023-08-22T23:22:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.906+0000 7fb29e7fc640 10 client.4363.objecter ms_dispatch 0x7fb2a8000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.906+0000 7fb29e7fc640 3 client.4363.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.906+0000 7fb29e7fc640 3 client.4363.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:36.907+0000 7fb29e7fc640 20 client.4363.objecter dump_active .. 0 homeless 2023-08-22T23:22:37.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.158+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fb2a0076770 con 0x7fb2a80b1380 2023-08-22T23:22:37.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.158+0000 7fb29e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.158+0000 7fb29e7fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:37.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.332+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2a813eca0 con 0x7fb28c06ba30 2023-08-22T23:22:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.334+0000 7fb29e7fc640 1 -- 172.21.15.29:0/1233187875 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48057 (secure 0 0 0) 0x7fb2a4001c90 con 0x7fb28c06ba30 2023-08-22T23:22:37.336 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:22:37.337 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:22:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 10 librados: watch_flush enter 2023-08-22T23:22:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 10 librados: watch_flush exit 2023-08-22T23:22:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 20 client.4363.objecter shutdown clearing up homeless session... 2023-08-22T23:22:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 10 client.4363.objecter successfully canceled tick 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb28c06ba30 msgr2=0x7fb28c06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb28c06ba30 0x7fb28c06def0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4001560 tx=0x7fb2a4009040 comp rx=0 tx=0).stop 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 10 monclient: shutdown 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 msgr2=0x7fb2a81d96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 0x7fb2a81d96e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00097d0 tx=0x7fb2a00046c0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 shutdown_connections 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb28c06ba30 0x7fb28c06def0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2a80b1380 0x7fb2a81d96e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2a80a7f70 0x7fb2a81d91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 --2- 172.21.15.29:0/1233187875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2a80a7b90 0x7fb2a813ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 >> 172.21.15.29:0/1233187875 conn(0x7fb2a80b6a00 msgr2=0x7fb2a813cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 shutdown_connections 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 -- 172.21.15.29:0/1233187875 wait complete. 2023-08-22T23:22:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.336+0000 7fb2afd0f640 1 librados: shutdown 2023-08-22T23:22:37.348 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-08-22T23:22:36.210141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218652,"kb_used_data":2456,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":754756068,"statfs":{"total":773094113280,"available":772870213632,"internally_reserved":0,"allocated":2514944,"data_stored":1374168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.591165"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.197751+0000","last_change":"2023-08-22T23:22:30.241386+0000","last_active":"2023-08-22T23:22:32.197751+0000","last_peered":"2023-08-22T23:22:32.197751+0000","last_clean":"2023-08-22T23:22:32.197751+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:22:32.197751+0000","last_undegraded":"2023-08-22T23:22:32.197751+0000","last_fullsized":"2023-08-22T23:22:32.197751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:44:37.084613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.777655+0000","last_change":"2023-08-22T23:22:30.276299+0000","last_active":"2023-08-22T23:22:32.777655+0000","last_peered":"2023-08-22T23:22:32.777655+0000","last_clean":"2023-08-22T23:22:32.777655+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:22:32.777655+0000","last_undegraded":"2023-08-22T23:22:32.777655+0000","last_fullsized":"2023-08-22T23:22:32.777655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-23T23:36:37.347305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.194575+0000","last_change":"2023-08-22T23:22:30.036499+0000","last_active":"2023-08-22T23:22:32.194575+0000","last_peered":"2023-08-22T23:22:32.194575+0000","last_clean":"2023-08-22T23:22:32.194575+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:22:32.194575+0000","last_undegraded":"2023-08-22T23:22:32.194575+0000","last_fullsized":"2023-08-22T23:22:32.194575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:38:38.054037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.188923+0000","last_change":"2023-08-22T23:22:30.442903+0000","last_active":"2023-08-22T23:22:32.188923+0000","last_peered":"2023-08-22T23:22:32.188923+0000","last_clean":"2023-08-22T23:22:32.188923+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:22:32.188923+0000","last_undegraded":"2023-08-22T23:22:32.188923+0000","last_fullsized":"2023-08-22T23:22:32.188923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:53:41.101184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.184640+0000","last_change":"2023-08-22T23:22:30.393248+0000","last_active":"2023-08-22T23:22:32.184640+0000","last_peered":"2023-08-22T23:22:32.184640+0000","last_clean":"2023-08-22T23:22:32.184640+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:22:32.184640+0000","last_undegraded":"2023-08-22T23:22:32.184640+0000","last_fullsized":"2023-08-22T23:22:32.184640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:16:05.115250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.747938+0000","last_change":"2023-08-22T23:22:31.576502+0000","last_active":"2023-08-22T23:22:32.747938+0000","last_peered":"2023-08-22T23:22:32.747938+0000","last_clean":"2023-08-22T23:22:32.747938+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:22:32.747938+0000","last_undegraded":"2023-08-22T23:22:32.747938+0000","last_fullsized":"2023-08-22T23:22:32.747938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:26:48.005613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.188906+0000","last_change":"2023-08-22T23:22:29.967457+0000","last_active":"2023-08-22T23:22:32.188906+0000","last_peered":"2023-08-22T23:22:32.188906+0000","last_clean":"2023-08-22T23:22:32.188906+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:22:32.188906+0000","last_undegraded":"2023-08-22T23:22:32.188906+0000","last_fullsized":"2023-08-22T23:22:32.188906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:34:10.033572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.201944+0000","last_change":"2023-08-22T23:22:30.329629+0000","last_active":"2023-08-22T23:22:32.201944+0000","last_peered":"2023-08-22T23:22:32.201944+0000","last_clean":"2023-08-22T23:22:32.201944+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:22:32.201944+0000","last_undegraded":"2023-08-22T23:22:32.201944+0000","last_fullsized":"2023-08-22T23:22:32.201944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:44:32.527595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'64","reported_seq":103,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.027633+0000","last_change":"2023-08-22T23:22:24.344727+0000","last_active":"2023-08-22T23:22:32.027633+0000","last_peered":"2023-08-22T23:22:32.027633+0000","last_clean":"2023-08-22T23:22:32.027633+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:22:32.027633+0000","last_undegraded":"2023-08-22T23:22:32.027633+0000","last_fullsized":"2023-08-22T23:22:32.027633+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_clean_scrub_stamp":"2023-08-22T23:22:21.305061+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:48:33.189247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":471588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.531000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.122}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.335000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.368}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.809000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.52}]}]},{"osd":3,"up_from":16,"seq":68719476743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0950000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.064}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.491}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.75}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.556000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.858000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.206}]}]},{"osd":2,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5939999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.393999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2489999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.024000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.516999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.632999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.558}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.574000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":67.614000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.718999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.403}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.132999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.689}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.542000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.102}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.974}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.780000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.934999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.103000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.998000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":33.354999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":68000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0210000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8979999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.935}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0920000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.358000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.457000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.617000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2939999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.702}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.463999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.25}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.550999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.048999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.425999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":471588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.457999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.183999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.087999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":82.183999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":92.167000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:22:37.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:22:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.459+0000 7fe77d857640 10 monclient: get_monmap_and_config 2023-08-22T23:22:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.459+0000 7fe77d857640 10 monclient: build_initial_monmap 2023-08-22T23:22:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.459+0000 7fe77d857640 10 monclient: monmap: 2023-08-22T23:22:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:37.460953+0000 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:37.460953+0000 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:37.463 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.460+0000 7fe77d857640 1 Processor -- start 2023-08-22T23:22:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.460+0000 7fe77d857640 1 -- start start 2023-08-22T23:22:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.460+0000 7fe77d857640 10 monclient: init 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7780a7b90 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780b1380 0x7fe7781417e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): picked mon.noname-b con 0x7fe7780b1380 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7780a7f70 0x7fe77814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): picked mon.noname-a con 0x7fe7780a7f70 addr v2:172.21.15.29:3300/0 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780a9b00 con 0x7fe7780a7b90 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7780a9de0 con 0x7fe7780a7f70 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7780aa0c0 con 0x7fe7780b1380 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe77d857640 10 monclient(hunting): authenticate will time out at 1521.711690s 2023-08-22T23:22:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.461+0000 7fe7777fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7780a7f70 0x7fe77814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7780a7f70 0x7fe77814ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33660/0 (socket says 172.21.15.29:33660) 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 1 -- 172.21.15.29:0/47968130 learned_addr learned my addr 172.21.15.29:0/47968130 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): get_auth_request con 0x7fe7780a7f70 auth_method 0 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.462+0000 7fe7777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2023-08-22T23:22:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 1 -- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780b1380 msgr2=0x7fe7781417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 1 --2- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780b1380 0x7fe7781417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 1 -- 172.21.15.29:0/47968130 >> v1:172.21.15.52:6789/0 conn(0x7fe7780a7b90 legacy=0x7fe77813e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 1 monclient: found mon.noname-a 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.464+0000 7fe7777fe640 1 -- 172.21.15.29:0/47968130 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7781412c0 con 0x7fe7780a7f70 2023-08-22T23:22:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.466+0000 7fe7777fe640 1 --2- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780a7f70 0x7fe77814ba30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe770009e50 tx=0x7fe77002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53173683e59ca9fc server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 1 -- 172.21.15.29:0/47968130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77002ca70 con 0x7fe7780a7f70 2023-08-22T23:22:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: dump: 2023-08-22T23:22:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: _finish_auth 0 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: _check_auth_tickets 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 1 -- 172.21.15.29:0/47968130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77002cbd0 con 0x7fe7780a7f70 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe77d857640 5 monclient: authenticate success, global_id 4374 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe77d857640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe77d857640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe77d857640 4 set_mon_vals no callback set 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 1 -- 172.21.15.29:0/47968130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7700343f0 con 0x7fe7780a7f70 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.470+0000 7fe7767fc640 10 monclient: dump: 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:37.476 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 10 monclient: shutdown 2023-08-22T23:22:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 -- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780a7f70 msgr2=0x7fe77814ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 --2- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780a7f70 0x7fe77814ba30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe770009e50 tx=0x7fe77002bed0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 -- 172.21.15.29:0/47968130 shutdown_connections 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 --2- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780a7f70 0x7fe77814ba30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 --2- 172.21.15.29:0/47968130 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780b1380 0x7fe7781417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 -- 172.21.15.29:0/47968130 >> 172.21.15.29:0/47968130 conn(0x7fe7780b6a00 msgr2=0x7fe7780b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 -- 172.21.15.29:0/47968130 shutdown_connections 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.471+0000 7fe77d857640 1 -- 172.21.15.29:0/47968130 wait complete. 2023-08-22T23:22:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.472+0000 7fe77d857640 10 monclient: build_initial_monmap 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.472+0000 7fe77d857640 10 monclient: monmap: 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.472+0000 7fe77d857640 1 librados: starting msgr at 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.472+0000 7fe77d857640 1 librados: starting objecter 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.473+0000 7fe77d857640 1 Processor -- start 2023-08-22T23:22:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.473+0000 7fe77d857640 1 -- start start 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.473+0000 7fe77d857640 1 librados: setting wanted keys 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.473+0000 7fe77d857640 1 librados: calling monclient init 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.473+0000 7fe77d857640 10 monclient: init 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7780a7b90 0x7fe778138e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): picked mon.noname-b con 0x7fe7780a7b90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780a7f70 0x7fe778132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): picked mon.noname-c con 0x7fe7780a7f70 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 0x7fe7781330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): picked mon.noname-a con 0x7fe7780b1380 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe778143b90 con 0x7fe7780b1380 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7780b6060 con 0x7fe7780a7b90 2023-08-22T23:22:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7780b6340 con 0x7fe7780a7f70 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe77d857640 10 monclient(hunting): authenticate will time out at 1521.724586s 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 0x7fe7781330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 0x7fe7781330a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33676/0 (socket says 172.21.15.29:33676) 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 1 -- 172.21.15.29:0/3892384110 learned_addr learned my addr 172.21.15.29:0/3892384110 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 10 monclient(hunting): get_auth_request con 0x7fe7780b1380 auth_method 0 2023-08-22T23:22:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.474+0000 7fe7777fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.475+0000 7fe7777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.475+0000 7fe7777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.475+0000 7fe7777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2023-08-22T23:22:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 1 -- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780a7f70 msgr2=0x7fe778132b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780a7f70 0x7fe778132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 1 -- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7780a7b90 msgr2=0x7fe778138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7780a7b90 0x7fe778138e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 10 monclient: _finish_hunting 0 2023-08-22T23:22:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 1 monclient: found mon.noname-a 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.478+0000 7fe7777fe640 1 -- 172.21.15.29:0/3892384110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770009b30 con 0x7fe7780b1380 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.480+0000 7fe7777fe640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 0x7fe7781330a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe770036040 tx=0x7fe770004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.480+0000 7fe774ff9640 10 client.?.objecter ms_handle_connect 0x7fe7780b1380 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.480+0000 7fe774ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77003e450 con 0x7fe7780b1380 2023-08-22T23:22:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: dump: 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 5 monclient: authenticate success, global_id 4377 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77003e5d0 con 0x7fe7780b1380 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 10 monclient: _renew_subs 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77003e980 con 0x7fe7780b1380 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7781335e0 con 0x7fe7780b1380 2023-08-22T23:22:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe775ffb640 4 set_mon_vals no callback set 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe774ff9640 10 monclient: dump: 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 10 monclient: _renew_subs 2023-08-22T23:22:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe778144f80 con 0x7fe7780b1380 2023-08-22T23:22:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.484+0000 7fe77d857640 1 librados: init done 2023-08-22T23:22:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.485+0000 7fe77d857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.485+0000 7fe77d857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.485+0000 7fe77d857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:22:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.485+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7780b6550 con 0x7fe7780b1380 2023-08-22T23:22:37.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.749+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe77003eae0 con 0x7fe7780b1380 2023-08-22T23:22:37.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.749+0000 7fe774ff9640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe75806ba30 0x7fe75806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:37.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.749+0000 7fe776ffd640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe75806ba30 0x7fe75806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:37.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.750+0000 7fe776ffd640 10 monclient: get_auth_request con 0x7fe75806ba30 auth_method 0 2023-08-22T23:22:37.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.750+0000 7fe774ff9640 10 client.4377.objecter ms_handle_connect 0x7fe75806ba30 2023-08-22T23:22:37.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.750+0000 7fe776ffd640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe75806ba30 0x7fe75806def0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe768001560 tx=0x7fe768007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:37.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe77007b490 con 0x7fe7780b1380 2023-08-22T23:22:37.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 10 client.4377.objecter ms_dispatch 0x7fe778000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:37.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 3 client.4377.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:37.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 3 client.4377.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:37.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 20 client.4377.objecter dump_active .. 0 homeless 2023-08-22T23:22:37.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fe770077140 con 0x7fe7780b1380 2023-08-22T23:22:37.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:37.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.763+0000 7fe774ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:37.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.937+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe77813eca0 con 0x7fe75806ba30 2023-08-22T23:22:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.939+0000 7fe774ff9640 1 -- 172.21.15.29:0/3892384110 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48057 (secure 0 0 0) 0x7fe7680050d0 con 0x7fe75806ba30 2023-08-22T23:22:37.941 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:22:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:22:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 10 librados: watch_flush enter 2023-08-22T23:22:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 10 librados: watch_flush exit 2023-08-22T23:22:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 20 client.4377.objecter shutdown clearing up homeless session... 2023-08-22T23:22:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 10 client.4377.objecter successfully canceled tick 2023-08-22T23:22:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe75806ba30 msgr2=0x7fe75806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe75806ba30 0x7fe75806def0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe768001560 tx=0x7fe768007990 comp rx=0 tx=0).stop 2023-08-22T23:22:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 10 monclient: shutdown 2023-08-22T23:22:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 msgr2=0x7fe7781330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 0x7fe7781330a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe770036040 tx=0x7fe770004a40 comp rx=0 tx=0).stop 2023-08-22T23:22:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 shutdown_connections 2023-08-22T23:22:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe75806ba30 0x7fe75806def0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7780b1380 0x7fe7781330a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe7780a7f70 0x7fe778132b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 --2- 172.21.15.29:0/3892384110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7780a7b90 0x7fe778138e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 >> 172.21.15.29:0/3892384110 conn(0x7fe7780b6a00 msgr2=0x7fe7781370c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 shutdown_connections 2023-08-22T23:22:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 -- 172.21.15.29:0/3892384110 wait complete. 2023-08-22T23:22:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:37.941+0000 7fe77d857640 1 librados: shutdown 2023-08-22T23:22:37.955 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-08-22T23:22:36.210141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218652,"kb_used_data":2456,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":754756068,"statfs":{"total":773094113280,"available":772870213632,"internally_reserved":0,"allocated":2514944,"data_stored":1374168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.591165"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.197751+0000","last_change":"2023-08-22T23:22:30.241386+0000","last_active":"2023-08-22T23:22:32.197751+0000","last_peered":"2023-08-22T23:22:32.197751+0000","last_clean":"2023-08-22T23:22:32.197751+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:22:32.197751+0000","last_undegraded":"2023-08-22T23:22:32.197751+0000","last_fullsized":"2023-08-22T23:22:32.197751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:44:37.084613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.777655+0000","last_change":"2023-08-22T23:22:30.276299+0000","last_active":"2023-08-22T23:22:32.777655+0000","last_peered":"2023-08-22T23:22:32.777655+0000","last_clean":"2023-08-22T23:22:32.777655+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:22:32.777655+0000","last_undegraded":"2023-08-22T23:22:32.777655+0000","last_fullsized":"2023-08-22T23:22:32.777655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-23T23:36:37.347305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.194575+0000","last_change":"2023-08-22T23:22:30.036499+0000","last_active":"2023-08-22T23:22:32.194575+0000","last_peered":"2023-08-22T23:22:32.194575+0000","last_clean":"2023-08-22T23:22:32.194575+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:22:32.194575+0000","last_undegraded":"2023-08-22T23:22:32.194575+0000","last_fullsized":"2023-08-22T23:22:32.194575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:38:38.054037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.188923+0000","last_change":"2023-08-22T23:22:30.442903+0000","last_active":"2023-08-22T23:22:32.188923+0000","last_peered":"2023-08-22T23:22:32.188923+0000","last_clean":"2023-08-22T23:22:32.188923+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:22:32.188923+0000","last_undegraded":"2023-08-22T23:22:32.188923+0000","last_fullsized":"2023-08-22T23:22:32.188923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:53:41.101184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.184640+0000","last_change":"2023-08-22T23:22:30.393248+0000","last_active":"2023-08-22T23:22:32.184640+0000","last_peered":"2023-08-22T23:22:32.184640+0000","last_clean":"2023-08-22T23:22:32.184640+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:22:32.184640+0000","last_undegraded":"2023-08-22T23:22:32.184640+0000","last_fullsized":"2023-08-22T23:22:32.184640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:16:05.115250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.747938+0000","last_change":"2023-08-22T23:22:31.576502+0000","last_active":"2023-08-22T23:22:32.747938+0000","last_peered":"2023-08-22T23:22:32.747938+0000","last_clean":"2023-08-22T23:22:32.747938+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:22:32.747938+0000","last_undegraded":"2023-08-22T23:22:32.747938+0000","last_fullsized":"2023-08-22T23:22:32.747938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:26:48.005613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.188906+0000","last_change":"2023-08-22T23:22:29.967457+0000","last_active":"2023-08-22T23:22:32.188906+0000","last_peered":"2023-08-22T23:22:32.188906+0000","last_clean":"2023-08-22T23:22:32.188906+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:22:32.188906+0000","last_undegraded":"2023-08-22T23:22:32.188906+0000","last_fullsized":"2023-08-22T23:22:32.188906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:34:10.033572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.201944+0000","last_change":"2023-08-22T23:22:30.329629+0000","last_active":"2023-08-22T23:22:32.201944+0000","last_peered":"2023-08-22T23:22:32.201944+0000","last_clean":"2023-08-22T23:22:32.201944+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:22:32.201944+0000","last_undegraded":"2023-08-22T23:22:32.201944+0000","last_fullsized":"2023-08-22T23:22:32.201944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:44:32.527595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'64","reported_seq":103,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-08-22T23:22:32.027633+0000","last_change":"2023-08-22T23:22:24.344727+0000","last_active":"2023-08-22T23:22:32.027633+0000","last_peered":"2023-08-22T23:22:32.027633+0000","last_clean":"2023-08-22T23:22:32.027633+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:22:32.027633+0000","last_undegraded":"2023-08-22T23:22:32.027633+0000","last_fullsized":"2023-08-22T23:22:32.027633+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_clean_scrub_stamp":"2023-08-22T23:22:21.305061+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:48:33.189247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":471588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.531000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.122}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.335000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.368}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.809000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.52}]}]},{"osd":3,"up_from":16,"seq":68719476743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0950000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.064}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.491}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.75}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.556000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.858000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.206}]}]},{"osd":2,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5939999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.393999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2489999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.024000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.516999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.632999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.558}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.574000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":67.614000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.718999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.403}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.132999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.689}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.542000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.102}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.974}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.780000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.934999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.103000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.998000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":33.354999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":68000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0210000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8979999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.935}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0920000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.358000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.457000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.617000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":73748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2939999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.702}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.463999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.25}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.550999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.048999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.425999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":471588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.457999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.183999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.087999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":82.183999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":92.167000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:22:37.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-08-22T23:22:37.957 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-08-22T23:22:37.957 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-08-22T23:22:37.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-08-22T23:22:38.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.061+0000 7f1eea580640 10 monclient: get_monmap_and_config 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.061+0000 7f1eea580640 10 monclient: build_initial_monmap 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.061+0000 7f1eea580640 10 monclient: monmap: 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:22:38.062400+0000 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:22:38.062400+0000 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:22:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:22:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:22:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:22:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:22:38.064 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.061+0000 7f1eea580640 1 Processor -- start 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.061+0000 7f1eea580640 1 -- start start 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.061+0000 7f1eea580640 10 monclient: init 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:38.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee40a8210 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ee40b01e0 addr v1:172.21.15.52:6789/0 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1ee40b0600 0x7f1ee4144da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee40b0600 addr v2:172.21.15.52:3300/0 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee40a7770 con 0x7f1ee40b01e0 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee41452e0 con 0x7f1ee40b0600 2023-08-22T23:22:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40a8b30 con 0x7f1ee40a8210 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.062+0000 7f1eea580640 10 monclient(hunting): authenticate will time out at 1522.312941s 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee37fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f1ee40b01e0 0x7f1ee4144690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44684/0 (socket says 172.21.15.29:44684) 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee37fe640 1 -- 172.21.15.29:0/2811301020 learned_addr learned my addr 172.21.15.29:0/2811301020 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee3fff640 10 monclient(hunting): get_auth_request con 0x7f1ee40a8210 auth_method 0 2023-08-22T23:22:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee8b1d640 1 --2- 172.21.15.29:0/2811301020 >> v2:172.21.15.52:3300/0 conn(0x7f1ee40b0600 0x7f1ee4144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee8b1d640 10 monclient(hunting): get_auth_request con 0x7f1ee40b0600 auth_method 0 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee8b1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee8b1d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.063+0000 7f1ee8b1d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.064+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.064+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.064+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.064+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.064+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.064+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2023-08-22T23:22:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 1 -- 172.21.15.29:0/2811301020 >> v2:172.21.15.52:3300/0 conn(0x7f1ee40b0600 msgr2=0x7f1ee4144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:22:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 1 --2- 172.21.15.29:0/2811301020 >> v2:172.21.15.52:3300/0 conn(0x7f1ee40b0600 0x7f1ee4144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 1 -- 172.21.15.29:0/2811301020 >> v1:172.21.15.52:6789/0 conn(0x7f1ee40b01e0 legacy=0x7f1ee4144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 10 monclient: _finish_hunting 0 2023-08-22T23:22:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 1 monclient: found mon.noname-b 2023-08-22T23:22:38.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:38.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:22:38.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.065+0000 7f1ee3fff640 1 -- 172.21.15.29:0/2811301020 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee40a8d30 con 0x7f1ee40a8210 2023-08-22T23:22:38.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.067+0000 7f1ee3fff640 1 --2- 172.21.15.29:0/2811301020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40a85f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40097a0 tx=0x7f1ed402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50fe1eeb7348cc40 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:38.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 1 -- 172.21.15.29:0/2811301020 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4031830 con 0x7f1ee40a8210 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: dump: 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: _finish_auth 0 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 5 monclient: authenticate success, global_id 4328 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 1 -- 172.21.15.29:0/2811301020 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed4031990 con 0x7f1ee40a8210 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 10 monclient: get_monmap_and_config success 2023-08-22T23:22:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 1 -- 172.21.15.29:0/2811301020 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4031d20 con 0x7f1ee40a8210 2023-08-22T23:22:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 4 set_mon_vals no callback set 2023-08-22T23:22:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:22:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1ee2ffd640 10 monclient: dump: 2023-08-22T23:22:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:38.078 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 10 monclient: shutdown 2023-08-22T23:22:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 1 -- 172.21.15.29:0/2811301020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 msgr2=0x7f1ee40a85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.071+0000 7f1eea580640 1 --2- 172.21.15.29:0/2811301020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40a85f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40097a0 tx=0x7f1ed402b8d0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 1 -- 172.21.15.29:0/2811301020 shutdown_connections 2023-08-22T23:22:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 1 --2- 172.21.15.29:0/2811301020 >> v2:172.21.15.52:3300/0 conn(0x7f1ee40b0600 0x7f1ee4144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 1 --2- 172.21.15.29:0/2811301020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40a85f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 1 -- 172.21.15.29:0/2811301020 >> 172.21.15.29:0/2811301020 conn(0x7f1ee40b6a10 msgr2=0x7f1ee40b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 1 -- 172.21.15.29:0/2811301020 shutdown_connections 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 1 -- 172.21.15.29:0/2811301020 wait complete. 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 10 monclient: build_initial_monmap 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.072+0000 7f1eea580640 10 monclient: monmap: 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:22:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 librados: starting msgr at 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 librados: starting objecter 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 Processor -- start 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 -- start start 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 librados: setting wanted keys 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 librados: calling monclient init 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 10 monclient: init 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 10 monclient: _reopen_session rank -1 2023-08-22T23:22:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee40a8210 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee40b01e0 0x7f1ee41d99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee40b01e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ee40b0600 0x7f1ee4142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.073+0000 7f1eea580640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee40b0600 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:22:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee413e560 con 0x7f1ee40b01e0 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee41425c0 con 0x7f1ee40b0600 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 10 monclient(hunting): start opening mon connection 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4143660 con 0x7f1ee40a8210 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 10 monclient(hunting): _renew_subs 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1eea580640 10 monclient(hunting): authenticate will time out at 1522.324497s 2023-08-22T23:22:38.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee40b01e0 0x7f1ee41d99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee8b1d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ee40b0600 0x7f1ee4142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee40b01e0 0x7f1ee41d99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33688/0 (socket says 172.21.15.29:33688) 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 1 -- 172.21.15.29:0/1439341725 learned_addr learned my addr 172.21.15.29:0/1439341725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 10 monclient(hunting): get_auth_request con 0x7f1ee40b01e0 auth_method 0 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee37fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee8b1d640 10 monclient(hunting): get_auth_request con 0x7f1ee40b0600 auth_method 0 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee8b1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee8b1d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee8b1d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee3fff640 10 monclient(hunting): get_auth_request con 0x7f1ee40a8210 auth_method 0 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee3fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:22:38.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.074+0000 7f1ee3fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:22:38.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.075+0000 7f1ee37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 1 -- 172.21.15.29:0/1439341725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 msgr2=0x7f1ee40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 1 -- 172.21.15.29:0/1439341725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee40b01e0 msgr2=0x7f1ee41d99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee40b01e0 0x7f1ee41d99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 10 monclient: _finish_hunting 0 2023-08-22T23:22:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 1 monclient: found mon.noname-b 2023-08-22T23:22:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:22:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:22:38.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.078+0000 7f1ee8b1d640 1 -- 172.21.15.29:0/1439341725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed4009400 con 0x7f1ee40b0600 2023-08-22T23:22:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.080+0000 7f1ee8b1d640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ee40b0600 0x7f1ee4142ae0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00eba0 tx=0x7f1edc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.080+0000 7f1ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f1ee40b0600 2023-08-22T23:22:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.080+0000 7f1ee0ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:22:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc00cd20 con 0x7f1ee40b0600 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: dump: 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: _finish_auth 0 2023-08-22T23:22:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc00f040 con 0x7f1ee40b0600 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0146c0 con 0x7f1ee40b0600 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1eea580640 5 monclient: authenticate success, global_id 4372 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:22:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee0ff9640 10 monclient: dump: 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:22:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 monclient: _renew_subs 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.084+0000 7f1ee27fc640 4 set_mon_vals no callback set 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee41442a0 con 0x7f1ee40b0600 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:22:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 monclient: _renew_subs 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee4143020 con 0x7f1ee40b0600 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 1 librados: init done 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.085+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee4143240 con 0x7f1ee40b0600 2023-08-22T23:22:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.346+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1edc014820 con 0x7f1ee40b0600 2023-08-22T23:22:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.347+0000 7f1ee0ff9640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1ec406ba60 0x7f1ec406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:22:38.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.347+0000 7f1ee3fff640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1ec406ba60 0x7f1ec406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:22:38.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.347+0000 7f1ee3fff640 10 monclient: get_auth_request con 0x7f1ec406ba60 auth_method 0 2023-08-22T23:22:38.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.347+0000 7f1ee3fff640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1ec406ba60 0x7f1ec406df20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4036000 tx=0x7f1ed403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:22:38.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.347+0000 7f1ee0ff9640 10 client.4372.objecter ms_handle_connect 0x7f1ec406ba60 2023-08-22T23:22:38.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.348+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1edc05c840 con 0x7f1ee40b0600 2023-08-22T23:22:38.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.348+0000 7f1ee0ff9640 10 client.4372.objecter ms_dispatch 0x7f1ee4000b90 osd_map(22..22 src has 1..22) v4 2023-08-22T23:22:38.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.348+0000 7f1ee0ff9640 3 client.4372.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:22:38.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.348+0000 7f1ee0ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:22:38.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.348+0000 7f1ee0ff9640 20 client.4372.objecter dump_active .. 0 homeless 2023-08-22T23:22:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.359+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f1edc054e70 con 0x7f1ee40b0600 2023-08-22T23:22:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.359+0000 7f1ee0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:22:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.359+0000 7f1ee0ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:22:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.594+0000 7f1eea580640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-08-22T23:22:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.594+0000 7f1eea580640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-08-22T23:22:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.594+0000 7f1eea580640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:22:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.594+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ee4066cb0 con 0x7f1ee40b0600 2023-08-22T23:22:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.601+0000 7f1ee0ff9640 1 -- 172.21.15.29:0/1439341725 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1edc058490 con 0x7f1ee40b0600 2023-08-22T23:22:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.601+0000 7f1ee0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-08-22T23:22:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.601+0000 7f1ee0ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:22:38.603 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:22:38.603 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-08-22T23:22:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 10 librados: watch_flush enter 2023-08-22T23:22:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 10 librados: watch_flush exit 2023-08-22T23:22:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 20 client.4372.objecter shutdown clearing up homeless session... 2023-08-22T23:22:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 10 client.4372.objecter successfully canceled tick 2023-08-22T23:22:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1ec406ba60 msgr2=0x7f1ec406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1ec406ba60 0x7f1ec406df20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4036000 tx=0x7f1ed403c040 comp rx=0 tx=0).stop 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 10 monclient: shutdown 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ee40b0600 msgr2=0x7f1ee4142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ee40b0600 0x7f1ee4142ae0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00eba0 tx=0x7f1edc00c490 comp rx=0 tx=0).stop 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 shutdown_connections 2023-08-22T23:22:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1ec406ba60 0x7f1ec406df20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ee40b0600 0x7f1ee4142ae0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee40b01e0 0x7f1ee41d99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 --2- 172.21.15.29:0/1439341725 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1ee40a8210 0x7f1ee40ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:22:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 >> 172.21.15.29:0/1439341725 conn(0x7f1ee40b6a10 msgr2=0x7f1ee40b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:22:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 shutdown_connections 2023-08-22T23:22:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 -- 172.21.15.29:0/1439341725 wait complete. 2023-08-22T23:22:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:22:38.603+0000 7f1eea580640 1 librados: shutdown 2023-08-22T23:22:38.616 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-08-22T23:22:38.616 INFO:teuthology.run_tasks:Running task workunit... 2023-08-22T23:22:38.632 INFO:tasks.workunit:Pulling workunits from ref f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T23:22:38.632 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-08-22T23:22:38.633 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2023-08-22T23:22:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:22:38.648 INFO:teuthology.orchestra.run.smithi029.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-08-22T23:22:38.648 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-08-22T23:22:38.703 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-08-22T23:22:38.704 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-08-22T23:22:38.759 INFO:tasks.workunit:timeout=6h 2023-08-22T23:22:38.759 INFO:tasks.workunit:cleanup=True 2023-08-22T23:22:38.760 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f0ed704638a5702969ce59fb4418946a60474648 2023-08-22T23:22:38.816 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-08-22T23:22:50.147 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:35.430 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.199 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:35.482 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.200 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:35.484 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.207 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:35.490 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.251 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:35.534 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.257 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:35.555 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.264 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:35.561 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.273 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:35.557 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:22:50.304 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:35.601 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.280 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:55.563 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.285 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:55.568 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.303 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:01:55.586 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.338 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:55.635 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.344 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:55.641 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.360 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:55.657 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.363 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:55.647 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.374 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:55.657 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:10.393 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:01:55.676 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:25.402 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to 'f0ed704638a5702969ce59fb4418946a60474648'. 2023-08-22T23:23:25.403 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.403 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-08-22T23:23:25.403 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2023-08-22T23:23:25.403 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2023-08-22T23:23:25.404 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.404 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2023-08-22T23:23:25.404 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2023-08-22T23:23:25.404 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.405 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2023-08-22T23:23:25.405 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.405 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2023-08-22T23:23:25.405 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.406 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2023-08-22T23:23:25.406 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.406 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-08-22T23:23:25.406 INFO:tasks.workunit.client.0.smithi029.stderr: 2023-08-22T23:23:25.406 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at f0ed704638a Merge branch 'wip-rf-map-update' of https://github.com/ronen-fr/ceph into wip-yuri10-testing-2023-08-17-1444 2023-08-22T23:23:25.407 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-08-22T23:23:25.476 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-08-22T23:23:25.478 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-08-22T23:23:25.478 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-08-22T23:23:26.499 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-08-22T23:23:26.621 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-08-22T23:23:26.662 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-08-22T23:23:26.664 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-08-22T23:23:26.664 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-08-22T23:23:26.704 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-08-22T23:23:26.707 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-08-22T23:23:26.707 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-08-22T23:23:26.764 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-08-22T23:23:26.765 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-08-22T23:23:26.765 DEBUG:teuthology.orchestra.run.smithi029:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f0ed704638a5702969ce59fb4418946a60474648 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-08-22T23:23:26.823 INFO:tasks.workunit.client.0.smithi029.stderr:+ parallel=1 2023-08-22T23:23:26.823 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' '' = --serial ']' 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:+ color= 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -t 1 ']' 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:+ declare -A pids 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_aio 2023-08-22T23:23:26.825 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_aio' 2023-08-22T23:23:26.825 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_aio 2023-08-22T23:23:26.826 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_aio 2023-08-22T23:23:26.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92188 2023-08-22T23:23:26.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_aio on pid 92188' 2023-08-22T23:23:26.827 INFO:tasks.workunit.client.0.smithi029.stdout:test api_aio on pid 92188 2023-08-22T23:23:26.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92188 2023-08-22T23:23:26.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-08-22T23:23:26.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.831 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_aio_pp 2023-08-22T23:23:26.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_aio_pp' 2023-08-22T23:23:26.831 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_aio_pp 2023-08-22T23:23:26.831 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.832 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.832 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.832 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.832 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.832 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.835 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.835 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.835 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.836 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.836 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_aio_pp 2023-08-22T23:23:26.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92195 2023-08-22T23:23:26.837 INFO:tasks.workunit.client.0.smithi029.stdout:test api_aio_pp on pid 92195 2023-08-22T23:23:26.840 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_aio_pp on pid 92195' 2023-08-22T23:23:26.840 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92195 2023-08-22T23:23:26.843 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-08-22T23:23:26.843 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.843 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.844 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_io 2023-08-22T23:23:26.844 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_io' 2023-08-22T23:23:26.844 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_io 2023-08-22T23:23:26.844 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.845 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.845 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.845 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.846 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.846 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.846 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.846 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.846 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.847 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.848 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.849 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.851 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.851 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.852 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.852 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.852 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.852 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.852 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_io 2023-08-22T23:23:26.853 INFO:tasks.workunit.client.0.smithi029.stdout:test api_io on pid 92204 2023-08-22T23:23:26.860 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92204 2023-08-22T23:23:26.860 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_io on pid 92204' 2023-08-22T23:23:26.861 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92204 2023-08-22T23:23:26.861 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.863 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.864 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-08-22T23:23:26.864 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_io_pp 2023-08-22T23:23:26.864 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_io_pp' 2023-08-22T23:23:26.865 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_io_pp 2023-08-22T23:23:26.865 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.865 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.865 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.866 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.866 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.866 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.867 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.867 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.867 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.868 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.868 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.868 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.868 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.869 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.869 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.869 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.870 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.870 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.870 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.870 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.871 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.871 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.871 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.872 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.872 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.872 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.872 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.873 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.873 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.873 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.874 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.874 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.874 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.874 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.875 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.875 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.875 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.875 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.876 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.876 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.876 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.876 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.877 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.877 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.877 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.877 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.878 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.878 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.878 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.878 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.879 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.879 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.879 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.879 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.880 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.880 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_io_pp 2023-08-22T23:23:26.880 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92216 2023-08-22T23:23:26.881 INFO:tasks.workunit.client.0.smithi029.stdout:test api_io_pp on pid 92216 2023-08-22T23:23:26.887 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_io_pp on pid 92216' 2023-08-22T23:23:26.887 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92216 2023-08-22T23:23:26.887 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.887 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_asio 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_asio' 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_asio 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.888 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.890 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.892 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.893 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.895 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.898 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.898 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:26.898 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.898 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:26.898 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:26.899 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_asio 2023-08-22T23:23:26.899 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:26.899 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92230 2023-08-22T23:23:26.899 INFO:tasks.workunit.client.0.smithi029.stdout:test api_asio on pid 92230 2023-08-22T23:23:26.903 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_asio on pid 92230' 2023-08-22T23:23:26.903 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92230 2023-08-22T23:23:26.903 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.903 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_list 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_list' 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_list 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:26.904 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.906 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.908 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.912 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.914 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.915 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.916 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.916 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.916 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.916 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.916 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.917 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.917 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92188/exe 2023-08-22T23:23:26.917 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_list 2023-08-22T23:23:26.917 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92247 2023-08-22T23:23:26.917 INFO:tasks.workunit.client.0.smithi029.stdout:test api_list on pid 92247 2023-08-22T23:23:26.921 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_list on pid 92247' 2023-08-22T23:23:26.921 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92247 2023-08-22T23:23:26.921 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.921 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.921 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-08-22T23:23:26.921 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_lock 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_lock' 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_lock 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:26.925 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.927 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.928 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_aio.log 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_aio 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:26.931 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:26.932 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.934 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.937 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:26.937 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.937 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:26.937 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:26.937 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_aio: /' 2023-08-22T23:23:26.937 INFO:tasks.workunit.client.0.smithi029.stdout:test api_lock on pid 92268 2023-08-22T23:23:26.939 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_lock 2023-08-22T23:23:26.939 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92195/exe 2023-08-22T23:23:26.939 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:26.939 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92268 2023-08-22T23:23:26.939 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_lock on pid 92268' 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92268 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_lock_pp 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_lock_pp' 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_lock_pp 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.941 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:26.946 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.948 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92204/exe 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.950 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.951 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_aio_pp: /' 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.952 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:26.954 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_aio_pp 2023-08-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-08-22T23:23:26.956 INFO:tasks.workunit.client.0.smithi029.stdout:test api_lock_pp on pid 92289 2023-08-22T23:23:26.958 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_lock_pp 2023-08-22T23:23:26.958 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92289 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_lock_pp on pid 92289' 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92289 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:26.960 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_misc 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_io.log 2023-08-22T23:23:26.961 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_misc' 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.962 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_io: /' 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.963 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:26.964 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.967 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_io 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92216/exe 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.968 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.969 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_misc 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:26.971 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:26.973 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.977 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_io_pp: /' 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:26.979 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_io_pp 2023-08-22T23:23:26.979 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:26.979 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:26.979 INFO:tasks.workunit.client.0.smithi029.stdout:test api_misc on pid 92317 2023-08-22T23:23:26.983 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:26.983 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:26.983 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:26.983 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.983 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_misc 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92317 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_misc on pid 92317' 2023-08-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92317 2023-08-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.991 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92247/exe 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-08-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:26.999 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_misc_pp 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_misc_pp' 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_misc_pp 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_list 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.002 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.008 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.009 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.009 INFO:tasks.workunit.client.0.smithi029.stdout:test api_misc_pp on pid 92341 2023-08-22T23:23:27.013 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_misc_pp 2023-08-22T23:23:27.013 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92341 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_misc_pp on pid 92341' 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92341 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92230/exe 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_list.log 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_list: /' 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_asio 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.017 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_tier_pp 2023-08-22T23:23:27.019 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_tier_pp' 2023-08-22T23:23:27.020 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.020 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.020 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.020 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.021 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.023 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.024 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_asio: /' 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_asio.log 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.027 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92268/exe 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92317/exe 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.030 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_lock 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_lock: /' 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_misc.log 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_misc: /' 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_lock.log 2023-08-22T23:23:27.032 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_tier_pp 2023-08-22T23:23:27.033 INFO:tasks.workunit.client.0.smithi029.stdout:test api_tier_pp on pid 92387 2023-08-22T23:23:27.036 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_tier_pp 2023-08-22T23:23:27.036 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92387 2023-08-22T23:23:27.036 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_tier_pp on pid 92387' 2023-08-22T23:23:27.036 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92387 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_misc 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_pool 2023-08-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_pool' 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.042 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_pool 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.043 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.046 INFO:tasks.workunit.client.0.smithi029.stdout:test api_pool on pid 92418 2023-08-22T23:23:27.047 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_pool 2023-08-22T23:23:27.047 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92418 2023-08-22T23:23:27.047 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_pool on pid 92418' 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92418 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.048 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_snapshots 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_snapshots' 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.049 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_snapshots 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.052 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.057 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92289/exe 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.061 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.061 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.061 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.061 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.061 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.062 INFO:tasks.workunit.client.0.smithi029.stdout:test api_snapshots on pid 92438 2023-08-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_snapshots 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92438 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_snapshots on pid 92438' 2023-08-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92438 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_lock_pp 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92341/exe 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.069 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_lock_pp: /' 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.071 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.072 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.072 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_misc_pp 2023-08-22T23:23:27.072 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-08-22T23:23:27.072 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.072 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.072 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.074 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.075 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_snapshots_pp 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_snapshots_pp' 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_snapshots_pp 2023-08-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi029.stdout:test api_snapshots_pp on pid 92470 2023-08-22T23:23:27.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_snapshots_pp 2023-08-22T23:23:27.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92470 2023-08-22T23:23:27.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_snapshots_pp on pid 92470' 2023-08-22T23:23:27.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92470 2023-08-22T23:23:27.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_stat 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.083 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_stat' 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92387/exe 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.087 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.088 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.089 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_tier_pp: /' 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_misc_pp: /' 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_tier_pp 2023-08-22T23:23:27.091 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.094 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.096 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.097 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.097 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.097 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_stat 2023-08-22T23:23:27.097 INFO:tasks.workunit.client.0.smithi029.stdout:test api_stat on pid 92499 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_stat 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92499 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_stat on pid 92499' 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92499 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.099 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_stat_pp 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_stat_pp' 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.104 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.105 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.105 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.105 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.105 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.105 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.105 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [----------] Global test environment set-up. 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [----------] 12 tests from AsioRados 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.952+0000 7f24c554a780 10 monclient: get_monmap_and_config 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.952+0000 7f24c554a780 10 monclient: build_initial_monmap 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.952+0000 7f24c554a780 10 monclient: monmap: 2023-08-22T23:23:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: epoch 0 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: last_changed 0.000000 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: created 0.000000 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: min_mon_release 0 (unknown) 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: election_strategy: 1 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:23:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.952+0000 7f24c554a780 1 Processor -- start 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.952+0000 7f24c554a780 1 -- start start 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.952+0000 7f24c554a780 10 monclient: init 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient: _reopen_session rank -1 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:23:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 0x5636bfb4e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): picked mon.noname-b con 0x7f24ac0030b0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636bfb4c6a0 0x5636bfaca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): picked mon.noname-a con 0x5636bfb4c6a0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb59090 0x5636bfb51eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): picked mon.noname-c con 0x5636bfb59090 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): start opening mon connection 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bfa41b70 con 0x5636bfb4c6a0 2023-08-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): start opening mon connection 2023-08-22T23:23:27.110 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bf9fc5c0 con 0x7f24ac0030b0 2023-08-22T23:23:27.110 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): start opening mon connection 2023-08-22T23:23:27.110 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636bf9fc260 con 0x5636bfb59090 2023-08-22T23:23:27.110 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): _renew_subs 2023-08-22T23:23:27.110 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.953+0000 7f24c554a780 10 monclient(hunting): authenticate will time out at 1571.203263s 2023-08-22T23:23:27.111 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.111 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.111 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.111 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.111 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.113 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.114 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.114 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.114 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.114 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.114 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.956+0000 7f24c3050640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 0x5636bfb4e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.957+0000 7f24c3050640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 0x5636bfb4e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37044/0 (socket says 172.21.15.29:37044) 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.957+0000 7f24c3050640 1 -- 172.21.15.29:0/1131666270 learned_addr learned my addr 172.21.15.29:0/1131666270 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.957+0000 7f24c3050640 10 monclient(hunting): get_auth_request con 0x7f24ac0030b0 auth_method 0 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.957+0000 7f24c3050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.957+0000 7f24c3050640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.957+0000 7f24c3050640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:23:27.115 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3851640 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb59090 0x5636bfb51eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3851640 10 monclient(hunting): get_auth_request con 0x5636bfb59090 auth_method 0 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3851640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3851640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3050640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.959+0000 7f24c3050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 1 -- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb59090 msgr2=0x5636bfb51eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb59090 0x5636bfb51eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 1 -- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636bfb4c6a0 msgr2=0x5636bfaca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636bfb4c6a0 0x5636bfaca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 10 monclient: _finish_hunting 0 2023-08-22T23:23:27.117 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 1 monclient: found mon.noname-b 2023-08-22T23:23:27.118 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.118 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.119 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_stat_pp 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.120 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.123 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.124 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_stat_pp 2023-08-22T23:23:27.124 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 20 monclient: _un_backoff reopen_interval_multipler nowtest api_stat_pp on pid 92554 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92554 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_stat_pp on pid 92554' 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92554 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.127 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_watch_notify 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_watch_notify' 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_watch_notify 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.132 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.134 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.135 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.136 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.136 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.136 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.136 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.136 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.136 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.137 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92438/exe 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.140 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.144 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.144 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.144 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.144 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.144 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.144 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92470/exe 2023-08-22T23:23:27.145 INFO:tasks.workunit.client.0.smithi029.stdout: 1 2023-08-22T23:23:27.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:23:27.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.961+0000 7f24c3050640 1 -- 172.21.15.29:0/1131666270 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bf9dd810 con 0x7f24ac0030b0 2023-08-22T23:23:27.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.970+0000 7f24c3050640 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 0x5636bfb4e010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00c450 tx=0x7f24bc00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936d467808a82974 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:23:27.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 1 -- 172.21.15.29:0/1131666270 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0115f0 con 0x7f24ac0030b0 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: dump: 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: epoch 1 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: min_mon_release 18 (reef) 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: election_strategy: 1 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: _finish_auth 0 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: _check_auth_tickets 2023-08-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 1 -- 172.21.15.29:0/1131666270 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc011750 con 0x7f24ac0030b0 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c554a780 5 monclient: authenticate success, global_id 4396 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c554a780 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 1 -- 172.21.15.29:0/1131666270 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc011ae0 con 0x7f24ac0030b0 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c184d640 10 monclient: dump: 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: epoch 1 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: min_mon_release 18 (reef) 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: election_strategy: 1 2023-08-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:23:27.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:23:27.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:23:27.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.151 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.151 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.152 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.153 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.155 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-08-22T23:23:27.156 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c554a780 10 monclient: get_monmap_and_config success 2023-08-22T23:23:27.156 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.975+0000 7f24c554a780 4 set_mon_vals no callback set 2023-08-22T23:23:27.156 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 10 monclient: shutdown 2023-08-22T23:23:27.156 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:23:27.156 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 -- 172.21.15.29:0/1131666270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 msgr2=0x5636bfb4e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 0x5636bfb4e010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00c450 tx=0x7f24bc00c910 comp rx=0 tx=0).stop 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 -- 172.21.15.29:0/1131666270 shutdown_connections 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb59090 0x5636bfb51eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636bfb4c6a0 0x5636bfaca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 --2- 172.21.15.29:0/1131666270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24ac0030b0 0x5636bfb4e010 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.976+0000 7f24c554a780 1 -- 172.21.15.29:0/1131666270 >> 172.21.15.29:0/1131666270 conn(0x5636bfacbba0 msgr2=0x5636bfb57c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 1 -- 172.21.15.29:0/1131666270 shutdown_connections 2023-08-22T23:23:27.157 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 1 -- 172.21.15.29:0/1131666270 wait complete. 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 10 monclient: build_initial_monmap 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 10 monclient: monmap: 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: epoch 0 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: last_changed 0.000000 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: created 0.000000 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: min_mon_release 0 (unknown) 2023-08-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: election_strategy: 1 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 1 librados: starting msgr at 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 1 librados: starting objecter 2023-08-22T23:23:27.159 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 1 Processor -- start 2023-08-22T23:23:27.160 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.977+0000 7f24c554a780 1 -- start start 2023-08-22T23:23:27.160 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 librados: setting wanted keys 2023-08-22T23:23:27.160 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 librados: calling monclient init 2023-08-22T23:23:27.160 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient: init 2023-08-22T23:23:27.160 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient: _reopen_session rank -1 2023-08-22T23:23:27.160 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:23:27.161 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.161 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.161 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.161 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.161 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.162 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_snapshots_pp 2023-08-22T23:23:27.163 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 --2- >> [v2:172.21.15.29:3301/test api_watch_notify on pid 92598 2023-08-22T23:23:27.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_watch_notify 2023-08-22T23:23:27.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92598 2023-08-22T23:23:27.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_watch_notify on pid 92598' 2023-08-22T23:23:27.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92598 2023-08-22T23:23:27.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.165 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.165 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.165 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.165 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.165 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.165 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_watch_notify_pp 2023-08-22T23:23:27.168 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_watch_notify_pp' 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_snapshots: /' 2023-08-22T23:23:27.169 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.170 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.171 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_watch_notify_pp 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_snapshots 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.172 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.173 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.174 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.175 INFO:tasks.workunit.client.0.smithi029.stdout:test api_watch_notify_pp on pid 92618 2023-08-22T23:23:27.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_watch_notify_pp 2023-08-22T23:23:27.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92618 2023-08-22T23:23:27.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_watch_notify_pp on pid 92618' 2023-08-22T23:23:27.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92618 2023-08-22T23:23:27.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.180 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.181 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.182 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.183 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.184 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.185 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.185 INFO:tasks.workunit.client.0.smithi029.stdout:0,v1:172.21.15.29:6790/0] conn(0x5636bfb4c6a0 0x5636bfb57f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:27.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): picked mon.noname-c con 0x5636bfb4c6a0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:23:27.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636bfb59090 0x5636bfbef850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:27.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): picked mon.noname-b con 0x5636bfb59090 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:23:27.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24ac0030b0 0x5636bfbefd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): picked mon.noname-a con 0x7f24ac0030b0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): start opening mon connection 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bf9be430 con 0x7f24ac0030b0 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): start opening mon connection 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bfbf03a0 con 0x5636bfb59090 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): start opening mon connection 2023-08-22T23:23:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636bfb554d0 con 0x5636bfb4c6a0 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): _renew_subs 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.978+0000 7f24c554a780 10 monclient(hunting): authenticate will time out at 1571.228385s 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.986+0000 7f24c284f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636bfb59090 0x5636bfbef850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.989+0000 7f24c284f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636bfb59090 0x5636bfbef850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:37068/0 (socket says 172.21.15.29:37068) 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.989+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 learned_addr learned my addr 172.21.15.29:0/534600864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.989+0000 7f24c284f640 10 monclient(hunting): get_auth_request con 0x5636bfb59090 auth_method 0 2023-08-22T23:23:27.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.989+0000 7f24c284f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:23:27.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.989+0000 7f24c284f640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:23:27.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.989+0000 7f24c284f640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:23:27.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.991+0000 7f24c3851640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24ac0030b0 0x5636bfbefd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:23:27.189 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.189 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.189 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.189 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.189 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_cmd 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_cmd' 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_cmd 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92418/exe 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.192 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.193 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.193 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.193 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23test api_cmd on pid 92646 2023-08-22T23:23:27.195 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.195 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.195 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.195 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_cmd 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.196 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92646 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_cmd on pid 92646' 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92646 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.198 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_pool: /' 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_pool.log 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.206 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92499/exe 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92554/exe 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.207 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_pool 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_stat.log 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_cmd_pp 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_cmd_pp' 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_stat 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_stat_pp 2023-08-22T23:23:27.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_stat_pp: /' 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.211 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.213 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_cmd_pp 2023-08-22T23:23:27.214 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_stat: /' 2023-08-22T23:23:27.214 INFO:tasks.workunit.client.0.smithi029.stdout:test api_cmd_pp on pid 92681 2023-08-22T23:23:27.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_cmd_pp 2023-08-22T23:23:27.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92681 2023-08-22T23:23:27.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_cmd_pp on pid 92681' 2023-08-22T23:23:27.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92681 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_service 2023-08-22T23:23:27.217 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.218 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.218 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.218 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.218 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.218 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.218 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.219 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.219 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_service' 2023-08-22T23:23:27.219 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.219 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.219 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.219 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.220 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.221 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_service 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.222 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.223 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.224 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.225 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.226 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.227 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.228 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.229 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.230 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.231 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92598/exe 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.232 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.233 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.234 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.235 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.235 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-08-22T23:23:27.235 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.235 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_watch_notify 2023-08-22T23:23:27.236 INFO:tasks.workunit.client.0.smithi029.stdout:test api_service on pid 92705 2023-08-22T23:23:27.238 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.238 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.238 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.238 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.238 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.239 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_service 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92705 2023-08-22T23:23:27.240 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_service on pid 92705' 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92705 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.241 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_watch_notify: /' 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.243 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_service_pp 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_service_pp' 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-08-22T23:23:27.246 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.248 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_service_pp 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.250 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.250 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.250 INFO:tasks.workunit.client.0.smithi029.stdout:test api_service_pp on pid 92741 2023-08-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_service_pp 2023-08-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92741 2023-08-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_service_pp on pid 92741' 2023-08-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92741 2023-08-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_c_write_operations 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_c_write_operations' 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.255 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.256 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.258 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.260 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92681/exe 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi029.stdout::23:26.991+0000 7f24c3851640 10 monclient(hunting): get_auth_request con 0x7f24ac0030b0 auth_method 0 2023-08-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.991+0000 7f24c3851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.991+0000 7f24c3851640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.991+0000 7f24c3851640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.992+0000 7f24c3851640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.992+0000 7f24c3851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.992+0000 7f24c3851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.993+0000 7f24c284f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:23:27.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.993+0000 7f24c284f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:26.993+0000 7f24c284f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 1 -- 172.21.15.29:0/534600864 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb4c6a0 msgr2=0x5636bfb57f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x5636bfb4c6a0 0x5636bfb57f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 1 -- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636bfb59090 msgr2=0x5636bfbef850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636bfb59090 0x5636bfbef850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:23:27.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 10 monclient: _finish_hunting 0 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 1 monclient: found mon.noname-a 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.000+0000 7f24c3851640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc00d040 con 0x7f24ac0030b0 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.031+0000 7f24c3851640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24ac0030b0 0x5636bfbefd90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x5636bfac8040 tx=0x7f24b8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.031+0000 7f24c084b640 10 client.?.objecter ms_handle_connect 0x7f24ac0030b0 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.031+0000 7f24c084b640 10 client.?.objecter resend_mon_ops 2023-08-22T23:23:27.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b80109a0 con 0x7f24ac0030b0 2023-08-22T23:23:27.267 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:23:27.267 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.267 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.267 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.267 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.269 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.270 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:23:27.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: dump: 2023-08-22T23:23:27.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: epoch 1 2023-08-22T23:23:27.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: min_mon_release 18 (reef) 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: election_strategy: 1 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: _finish_auth 0 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: _check_auth_tickets 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b8010b20 con 0x7f24ac0030b0 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b8010ed0 con 0x7f24ac0030b0 2023-08-22T23:23:27.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c084b640 10 monclient: dump: 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: epoch 1 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: min_mon_release 18 (reef) 2023-08-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: election_strategy: 1 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c184d640 4 set_mon_vals no callback set 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 5 monclient: authenticate success, global_id 4416 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 10 monclient: _renew_subs 2023-08-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:27.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5636bf9dd270 con 0x7f24ac0030b0 2023-08-22T23:23:27.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:23:27.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 10 monclient: _renew_subs 2023-08-22T23:23:27.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:27.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bfb55a50 con 0x7f24ac0030b0 2023-08-22T23:23:27.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.049+0000 7f24c554a780 1 librados: init done 2023-08-22T23:23:27.277 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.278 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.279 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.280 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92618/exe 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.283 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.285 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_c_write_operations 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.286 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.287 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.289 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.290 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_cmd_pp 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.292 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.293 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.294 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-08-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T2test api_c_write_operations on pid 92792 2023-08-22T23:23:27.298 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_c_write_operations 2023-08-22T23:23:27.298 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92792 2023-08-22T23:23:27.298 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_c_write_operations on pid 92792' 2023-08-22T23:23:27.298 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92792 2023-08-22T23:23:27.298 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92646/exe 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-08-22T23:23:27.300 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_c_read_operations 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_c_read_operations' 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.301 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.302 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_c_read_operations 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_cmd.log 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.307 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.308 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.309 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.309 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_cmd 2023-08-22T23:23:27.309 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.309 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:test api_c_read_operations on pid 92819 2023-08-22T23:23:27.310 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_c_read_operations 2023-08-22T23:23:27.310 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.310 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92819 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_c_read_operations on pid 92819' 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92819 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s api_cls_remote_reads 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' api_cls_remote_reads' 2023-08-22T23:23:27.311 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.312 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo api_cls_remote_reads 2023-08-22T23:23:27.312 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_cmd: /' 2023-08-22T23:23:27.312 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=api_cls_remote_reads 2023-08-22T23:23:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:test api_cls_remote_reads on pid 92830 2023-08-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92830 2023-08-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test api_cls_remote_reads on pid 92830' 2023-08-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92830 2023-08-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s list_parallel 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' list_parallel' 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.316 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.318 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo list_parallel 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=list_parallel 2023-08-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi029.stdout:test list_parallel on pid 92849 2023-08-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92849 2023-08-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test list_parallel on pid 92849' 2023-08-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92849 2023-08-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s open_pools_parallel 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' open_pools_parallel' 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo open_pools_parallel 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.327 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.328 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.329 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.330 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.332 INFO:tasks.workunit.client.0.smithi029.stdout:test open_pools_parallel on pid 92858 2023-08-22T23:23:27.333 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.333 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=open_pools_parallel 2023-08-22T23:23:27.333 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92858 2023-08-22T23:23:27.334 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test open_pools_parallel on pid 92858' 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92858 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.335 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:++ printf %25s delete_pools_parallel 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=' delete_pools_parallel' 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.337 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.338 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.339 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.340 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.341 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.342 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:++ echo delete_pools_parallel 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.343 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.344 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.344 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.344 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.344 INFO:tasks.workunit.client.0.smithi029.stderr:++ awk '{print $1}' 2023-08-22T23:23:27.344 INFO:tasks.workunit.client.0.smithi029.stdout:test delete_pools_parallel on pid 92873 2023-08-22T23:23:27.354 INFO:tasks.workunit.client.0.smithi029.stderr:+ ff=delete_pools_parallel 2023-08-22T23:23:27.354 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92873 2023-08-22T23:23:27.354 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'test delete_pools_parallel on pid 92873' 2023-08-22T23:23:27.354 INFO:tasks.workunit.client.0.smithi029.stderr:+ pids[$f]=92873 2023-08-22T23:23:27.354 INFO:tasks.workunit.client.0.smithi029.stderr:+ ret=0 2023-08-22T23:23:27.354 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 1 -eq 1 ']' 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92387 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92387 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.356 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.357 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.359 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.360 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.361 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.362 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.363 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.365 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.366 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.368 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.368 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.368 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.368 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.368 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.368 INFO:tasks.workunit.client.0.smithi029.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.369 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.371 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.374 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.375 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.379 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.380 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.381 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.382 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92830/exe 2023-08-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -f /etc/bashrc ']' 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.385 INFO:tasks.workunit.client.0.smithi029.stderr:+ . /etc/bashrc 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -z '' ']' 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:++ BASHRCSOURCED=Y 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi029.stderr:++ shopt -q login_shell 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:+++ umask 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.387 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' 0022 -eq 0 ']' 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:++ SHELL=/bin/bash 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92792/exe 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/bash_completion.sh 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/cobbler.sh 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorgrep.sh 2023-08-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92741/exe 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_c_write_operations 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-08-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.394 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.395 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.395 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.395 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.395 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.395 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.395 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_service_pp 2023-08-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_service_pp: /' 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92849/exe 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.398 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.399 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_list_parallel 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_list_parallel.log 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ list_parallel: /' 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92705/exe 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.400 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.401 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.402 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.403 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_service 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_service.log 2023-08-22T23:23:27.405 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'grep=grep --color=auto' 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'egrep=egrep --color=auto' 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorls.sh 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' '!' -t 0 ']' 2023-08-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi029.stderr:+++ return 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export S_COLORS=auto 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:+++ S_COLORS=auto 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.407 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-08-22T23:23:27.408 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:+++ /usr/libexec/grepconf.sh -c 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_service: /' 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.409 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.414 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/debuginfod.sh 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/gawk.sh 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-08-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/lang.sh 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/locale 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.418 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92819/exe 2023-08-22T23:23:27.420 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG_backup=en_US.UTF-8 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /etc/locale.conf ']' 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -x /usr/bin/sed ']' 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-08-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.422 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:++++ export LANG=en_US.UTF-8 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:++++ LANG=en_US.UTF-8 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-08-22T23:23:27.423 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:+++ LANG=en_US.UTF-8 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:+++ unset LANG_backup config 2023-08-22T23:23:27.424 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n '' ']' 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-08-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' dumb = linux ']' 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/less.sh 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:++ for i in /etc/profile.d/*.sh 2023-08-22T23:23:27.426 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:++ '[' '' ']' 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:++ . /etc/profile.d/which2.sh 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_api_c_read_operations 2023-08-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92873/exe 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.429 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:+++++ readlink /proc/92858/exe 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-08-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi029.stderr:++++ basename /usr/bin/bash 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_declare='declare -f' 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:+++ which_opt=-f 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export which_declare 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:+++ export -f which 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset i 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:++ unset -f pathmunge 2023-08-22T23:23:27.431 INFO:tasks.workunit.client.0.smithi029.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ export PATH 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ unset rc 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_delete_pools_parallel 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-08-22T23:23:27.432 INFO:tasks.workunit.client.0.smithi029.stderr:+ ceph_test_rados_open_pools_parallel 2023-08-22T23:23:28.197 INFO:tasks.workunit.client.0.smithi029.stdout:3:23:27.049+0000 7f24c554a780 10 librados: wait_for_osdmap waiting 2023-08-22T23:23:28.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.137+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f24b801d410 con 0x7f24ac0030b0 2023-08-22T23:23:28.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.138+0000 7f24c084b640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f24b0069c20 0x7f24b006c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:28.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.138+0000 7f24c3050640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f24b0069c20 0x7f24b006c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:23:28.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.138+0000 7f24c3050640 10 monclient: get_auth_request con 0x7f24b0069c20 auth_method 0 2023-08-22T23:23:28.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.140+0000 7f24c3050640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f24b0069c20 0x7f24b006c0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f24bc002230 tx=0x7f24bc008000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:23:28.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.140+0000 7f24c084b640 10 client.4416.objecter ms_handle_connect 0x7f24b0069c20 2023-08-22T23:23:28.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f24b8058ff0 con 0x7f24ac0030b0 2023-08-22T23:23:28.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(22..22 src has 1..22) v4 2023-08-22T23:23:28.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [22,22] > 0 2023-08-22T23:23:28.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding full epoch 22 2023-08-22T23:23:28.200 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:23:28.200 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c554a780 10 librados: wait_for_osdmap done waiting 2023-08-22T23:23:28.200 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c554a780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:28.201 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.150+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f24b400a040 con 0x7f24ac0030b0 2023-08-22T23:23:28.201 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.202+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24b8054ca0 con 0x7f24ac0030b0 2023-08-22T23:23:28.202 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.202+0000 7f24c084b640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-08-22T23:23:28.202 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.203+0000 7f24c104c640 10 client.4416.objecter _get_latest_version latest 22, have it 2023-08-22T23:23:28.202 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.203+0000 7f24c554a780 10 client.4416.objecter create_pool name=ceph_test_rados_api_asio 2023-08-22T23:23:28.203 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.203+0000 7f24c554a780 10 client.4416.objecter pool_op_submit 1 2023-08-22T23:23:28.203 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.203+0000 7f24c554a780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:28.203 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:27.203+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x5636bfa4b650 con 0x7f24ac0030b0 2023-08-22T23:23:28.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v23) v1 ==== 43+0+0 (secure 0 0 0) 0x7f24b8018020 con 0x7f24ac0030b0 2023-08-22T23:23:28.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 pool_op_reply(tid 1 (0) Success v23) v1 2023-08-22T23:23:28.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 client.4416.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) v1 2023-08-22T23:23:28.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 client.4416.objecter have request 1 at 0x5636bfb557b0 Op: create 2023-08-22T23:23:28.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 20 client.4416.objecter waiting for client to reach epoch 23 before calling back 2023-08-22T23:23:28.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:23:28.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:23:28.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:28.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7f24b006f960 con 0x7f24ac0030b0 2023-08-22T23:23:28.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.196+0000 7f24c084b640 10 client.4416.objecter done 2023-08-22T23:23:28.294 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3598+0+0 (secure 0 0 0) 0x7f24b805c720 con 0x7f24ac0030b0 2023-08-22T23:23:28.294 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(23..23 src has 1..23) v4 2023-08-22T23:23:28.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [23,23] > 22 2023-08-22T23:23:28.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 23 2023-08-22T23:23:28.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:23:28.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 10 librados: set snap read head -> 1 2023-08-22T23:23:28.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 10 librados: writefull oid=exist nspace= 2023-08-22T23:23:28.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55d00 2023-08-22T23:23:28.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 23 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:28.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:28.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:23:28.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x5636bfb4b7e0 0x5636bfb4bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:23:28.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:23:28.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [writefull 0~5 in=5b] tid 2 osd.6 2023-08-22T23:23:28.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:23:28.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 2 2023-08-22T23:23:28.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 15 client.4416.objecter _send_op 2 to 3.1 on osd.6 2023-08-22T23:23:30.706 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:15.989 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.708 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:15.991 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.715 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:16.012 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.716 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:15.999 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.734 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:16.017 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.743 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:16.040 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.765 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:16.049 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.799 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:16.096 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:30.801 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:16.084 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:36.980 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) v8 -- 0x5636bfbf8da0 con 0x5636bfb4b7e0 2023-08-22T23:23:36.980 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.289+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.322+0000 7f24c284f640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x5636bfb4b7e0 0x5636bfb4bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.322+0000 7f24c284f640 10 monclient: get_auth_request con 0x5636bfb4b7e0 auth_method 0 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.324+0000 7f24c084b640 10 client.4416.objecter ms_handle_connect 0x5636bfb4b7e0 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:28.324+0000 7f24c284f640 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x5636bfb4b7e0 0x5636bfb4bbc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:29.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:29.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:29.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:29.978+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:30.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:30.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:30.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:31.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:31.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:31.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:32.050+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:32.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:32.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:32.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:33.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:33.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:33.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:34.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:34.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:34.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:35.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:35.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:35.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:36.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:36.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:36.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:42.137 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:36.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:42.137 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:37.050+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:23:42.137 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:37.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:42.137 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:37.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:42.138 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:37.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:42.138 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:38.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:42.138 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:38.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:42.138 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:38.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:39.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:39.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:39.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:39.978+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:40.978+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:40.978+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:40.978+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:42.139 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:41.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:42.140 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:41.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:42.140 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:41.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:42.140 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:23:42.140 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:23:42.140 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:23:42.141 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:23:42.141 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:42.141 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=24}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:23:42.141 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.051+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2494003320 con 0x5636bfb4b7e0 2023-08-22T23:23:42.141 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.134+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 9 ==== osd_map(24..32 src has 1..32) v4 ==== 21311+0+0 (secure 0 0 0) 0x7f24b8060670 con 0x7f24ac0030b0 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.134+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(24..32 src has 1..32) v4 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.134+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [24,32] > 23 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.134+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 24 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 24 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.142 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 25 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 25 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 26 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 26 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.143 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 27 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 27 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 28 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 28 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.144 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 29 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 29 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 30 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 30 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.145 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 31 2023-08-22T23:23:42.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:42.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 31 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:42.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:42.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 32 2023-08-22T23:23:42.146 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:47.179 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 32 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:47.179 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:47.179 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:23:47.179 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.135+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:23:47.180 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:47.180 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:47.180 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:42.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:47.180 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:43.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:47.181 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:43.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:47.181 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:43.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:47.182 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:44.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:47.182 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:44.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:47.182 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:44.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:47.182 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:45.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:47.182 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:45.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:45.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:46.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:46.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:46.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:23:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:23:47.184 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:47.184 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=33}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:23:47.184 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.052+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2494003540 con 0x5636bfb4b7e0 2023-08-22T23:23:47.184 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 10 ==== osd_map(33..36 src has 1..36) v4 ==== 2954+0+0 (secure 0 0 0) 0x7f24b8058340 con 0x7f24ac0030b0 2023-08-22T23:23:47.184 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(33..36 src has 1..36) v4 2023-08-22T23:23:47.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [33,36] > 32 2023-08-22T23:23:47.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 33 2023-08-22T23:23:47.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:47.185 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 33 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:50.861 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:36.145 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:50.930 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:36.213 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.001 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:36.298 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.004 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:36.288 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.008 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:36.305 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.011 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:36.294 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.125 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:36.408 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.161 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:36.458 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:51.202 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:36.485 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:23:52.054 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:52.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 34 2023-08-22T23:23:52.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:52.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 34 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:52.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:52.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 35 2023-08-22T23:23:52.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 35 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 36 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 36 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:52.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.177+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:47.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:48.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:48.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:48.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:52.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:49.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:49.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:49.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:49.979+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:50.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:50.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:50.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:51.979+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:52.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:51.979+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:52.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:51.979+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:52.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:23:52.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:23:53.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:23:53.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:23:53.982 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=37}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.053+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f24940037e0 con 0x5636bfb4b7e0 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 11 ==== osd_map(37..40 src has 1..40) v4 ==== 3262+0+0 (secure 0 0 0) 0x7f24b8055070 con 0x7f24ac0030b0 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(37..40 src has 1..40) v4 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [37,40] > 36 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 37 2023-08-22T23:23:53.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 37 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 38 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 38 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:53.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 39 2023-08-22T23:23:53.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:53.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 39 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:53.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:53.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 40 2023-08-22T23:23:53.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:53.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 40 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:53.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:53.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:23:53.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.057+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:23:53.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:53.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:53.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:52.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:53.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:53.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:55.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2023-08-22T23:23:55.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [----------] Global test environment set-up. 2023-08-22T23:23:55.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [----------] 4 tests from LibRadosService 2023-08-22T23:23:55.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-08-22T23:23:55.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5265 ms) 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ OK ] LibRadosService.RegisterLate (294 ms) 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: cluster: 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: id: cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: health: HEALTH_WARN 2023-08-22T23:23:55.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 16 pool(s) do not have an application enabled 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: services: 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: mgr: x(active, since 115s), standbys: y 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: osd: 8 osds: 8 up (since 71s), 8 in (since 115s) 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-08-22T23:23:55.928 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: data: 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: pools: 26 pools, 777 pgs 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: objects: 2 objects, 449 KiB 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: pgs: 83.140% pgs unknown 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 15.701% pgs not active 2023-08-22T23:23:55.929 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 646 unknown 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 122 creating+peering 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 9 active+clean 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: cluster: 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: id: cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: health: HEALTH_WARN 2023-08-22T23:23:55.930 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 10 pool(s) do not have an application enabled 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: services: 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: mgr: x(active, since 117s), standbys: y 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 117s) 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-08-22T23:23:55.931 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: data: 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: pools: 29 pools, 849 pgs 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: objects: 3 objects, 449 KiB 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: pgs: 50.883% pgs unknown 2023-08-22T23:23:55.932 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 44.170% pgs not active 2023-08-22T23:23:55.933 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 432 unknown 2023-08-22T23:23:55.933 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 304 creating+peering 2023-08-22T23:23:55.933 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 71 creating+activating 2023-08-22T23:23:55.934 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 42 active+clean 2023-08-22T23:23:55.934 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.934 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: io: 2023-08-22T23:23:55.934 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: client: 217 B/s wr, 0 op/s rd, 0 op/s wr 2023-08-22T23:23:55.934 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3142 ms) 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ RUN ] LibRadosService.Status 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ OK ] LibRadosService.Status (20061 ms) 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [----------] 4 tests from LibRadosService (28762 ms total) 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [----------] Global test environment tear-down 2023-08-22T23:23:55.935 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [==========] 4 tests from 1 test suite ran. (28762 ms total) 2023-08-22T23:23:55.936 INFO:tasks.workunit.client.0.smithi029.stdout: api_service: [ PASSED ] 4 tests. 2023-08-22T23:23:57.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:53.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:53.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:54.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:54.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:54.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:55.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:55.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:55.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 1 ==== osd_map(24..42 src has 1..42) v4 ==== 28718+0+0 (crc 0 0 0) 0x7f24b4057700 con 0x5636bfb4b7e0 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(24..42 src has 1..42) v4 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [24,42] > 40 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 41 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 41 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 42 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 42 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.913+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:23:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:56.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:23:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=43}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:23:58.029 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.054+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2 api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-08-22T23:23:58.029 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [----------] Global test environment set-up. 2023-08-22T23:23:58.029 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-08-22T23:23:58.029 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-08-22T23:23:58.029 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5290 ms) 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (253 ms) 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20124 ms) 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: attempt 0 of 20 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5203 ms) 2023-08-22T23:23:58.030 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30870 ms total) 2023-08-22T23:23:58.031 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: 2023-08-22T23:23:58.031 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [----------] Global test environment tear-down 2023-08-22T23:23:58.031 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30870 ms total) 2023-08-22T23:23:58.031 INFO:tasks.workunit.client.0.smithi029.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-08-22T23:23:58.553 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:23:58.489+0000 1225d640 -1 osd.6 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:23:59.756 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:23:59.631+0000 1225d640 -1 osd.6 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:00.729 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:00.677+0000 1225d640 -1 osd.6 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:02.056 INFO:tasks.workunit.client.0.smithi029.stdout:4940037e0 con 0x5636bfb4b7e0 2023-08-22T23:24:02.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 12 ==== osd_map(43..44 src has 1..44) v4 ==== 2038+0+0 (secure 0 0 0) 0x7f24b8058690 con 0x7f24ac0030b0 2023-08-22T23:24:02.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(43..44 src has 1..44) v4 2023-08-22T23:24:02.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [43,44] > 42 2023-08-22T23:24:02.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 43 2023-08-22T23:24:02.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:02.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 43 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 44 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 44 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.178+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:02.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:02.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:02.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:57.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:02.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:58.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:02.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:58.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:02.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:58.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:02.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:59.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:02.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:59.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:02.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:59.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:02.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:23:59.980+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:24:02.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:00.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:02.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:00.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:02.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:00.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:02.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:01.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:02.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:01.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:02.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:01.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:02.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:02.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:02.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:02.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:24:02.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:02.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=45}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:24:02.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.055+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2494003970 con 0x5636bfb4b7e0 2023-08-22T23:24:02.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 13 ==== osd_map(45..48 src has 1..48) v4 ==== 8273+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:02.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(45..48 src has 1..48) v4 2023-08-22T23:24:02.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [45,48] > 44 2023-08-22T23:24:02.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 45 2023-08-22T23:24:02.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:02.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 45 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:02.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:02.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 46 2023-08-22T23:24:02.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:02.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 46 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:02.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.059+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:02.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 47 2023-08-22T23:24:02.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:02.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 47 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:02.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:02.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 48 2023-08-22T23:24:02.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 48 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=49}) v3 -- 0x7f24b0072520 con 0x7f24ac0030b0 2023-08-22T23:24:02.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:02.088 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:02.032+0000 1225d640 -1 osd.6 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:03.262 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:03.246+0000 1225d640 -1 osd.6 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:03.685 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.060+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:03.685 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 14 ==== osd_map(49..49 src has 1..49) v4 ==== 2101+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:03.686 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(49..49 src has 1..49) v4 2023-08-22T23:24:03.686 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [49,49] > 48 2023-08-22T23:24:03.686 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 49 2023-08-22T23:24:03.686 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:03.686 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 49 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=50}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:03.687 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.570+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.980+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.980+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:02.980+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 15 ==== osd_map(50..50 src has 1..50) v4 ==== 1395+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(50..50 src has 1..50) v4 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [50,50] > 49 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 50 2023-08-22T23:24:03.688 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:03.689 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 50 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:03.689 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:03.691 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:03.691 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:03.691 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:03.691 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=51}) v3 -- 0x7f24b007a0b0 con 0x7f24ac0030b0 2023-08-22T23:24:04.519 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:04.404+0000 1225d640 -1 osd.6 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:04.992 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:04.982+0000 1225d640 -1 osd.2 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:05.812 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:05.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.683+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:05.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:05.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:05.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:03.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:05.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 16 ==== osd_map(51..51 src has 1..51) v4 ==== 1316+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:05.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(51..51 src has 1..51) v4 2023-08-22T23:24:05.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [51,51] > 50 2023-08-22T23:24:05.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 51 2023-08-22T23:24:05.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:05.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 51 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:05.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:05.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:05.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:05.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:05.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=52}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:05.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:05.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.676+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:05.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:05.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:05.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:04.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:05.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.810+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 17 ==== osd_map(52..52 src has 1..52) v4 ==== 1023+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:05.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.810+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(52..52 src has 1..52) v4 2023-08-22T23:24:05.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.810+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [52,52] > 51 2023-08-22T23:24:05.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.810+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 52 2023-08-22T23:24:05.822 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:05.822 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 52 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:05.823 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:05.823 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:05.823 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:05.887 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:05.846+0000 1225d640 -1 osd.6 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:05.983 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:05.969+0000 1225d640 -1 osd.2 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:07.024 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:07.017+0000 1225d640 -1 osd.2 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:07.126 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:07.046+0000 1225d640 -1 osd.6 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:07.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:07.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=53}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:07.388 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:07.388 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.811+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:07.388 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:07.388 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:07.388 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:05.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:07.389 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 18 ==== osd_map(53..53 src has 1..53) v4 ==== 324+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:07.389 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(53..53 src has 1..53) v4 2023-08-22T23:24:07.389 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [53,53] > 52 2023-08-22T23:24:07.389 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 53 2023-08-22T23:24:07.389 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:07.389 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 53 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=54}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.177+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:07.390 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:06.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.056+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.056+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.056+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.056+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2494002f90 con 0x5636bfb4b7e0 2023-08-22T23:24:07.391 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 19 ==== osd_map(54..54 src has 1..54) v4 ==== 312+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:08.147 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:08.140+0000 1225d640 -1 osd.2 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:08.265 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:08.240+0000 1225d640 -1 osd.6 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:08.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(54..54 src has 1..54) v4 2023-08-22T23:24:08.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [54,54] > 53 2023-08-22T23:24:08.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 54 2023-08-22T23:24:08.983 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 54 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=55}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:08.984 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.385+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:07.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 20 ==== osd_map(55..55 src has 1..55) v4 ==== 700+0+0 (secure 0 0 0) 0x7f24b80874f0 con 0x7f24ac0030b0 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(55..55 src has 1..55) v4 2023-08-22T23:24:08.985 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [55,55] > 54 2023-08-22T23:24:08.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 55 2023-08-22T23:24:08.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 55 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=56}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:08.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:08.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.776+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:08.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:08.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:09.283 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:09.186+0000 1225d640 -1 osd.2 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:09.982 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:09.908+0000 1225d640 -1 osd.6 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:10.288 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:10.266+0000 1225d640 -1 osd.2 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:10.841 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:08.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:10.841 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 21 ==== osd_map(56..56 src has 1..56) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:10.841 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(56..56 src has 1..56) v4 2023-08-22T23:24:10.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [56,56] > 55 2023-08-22T23:24:10.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 56 2023-08-22T23:24:10.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:10.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 56 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:10.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:10.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:10.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:10.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:10.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=57}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.760+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:09.981+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:24:10.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 22 ==== osd_map(57..57 src has 1..57) v4 ==== 682+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(57..57 src has 1..57) v4 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [57,57] > 56 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 57 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 57 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:10.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:10.846 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 10 monclient: _renew_subs 2023-08-22T23:24:10.846 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:11.250 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:11.233+0000 1225d640 -1 osd.6 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:11.297 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:11.292+0000 1225d640 -1 osd.2 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:11.452 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:56.735 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.484 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:56.767 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.491 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:56.774 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.536 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:56.833 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.545 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:56.842 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.564 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:56.848 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.694 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:56.990 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.705 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:02:56.988 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:11.753 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:02:57.036 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:12.337 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:12.282+0000 1225d640 -1 osd.6 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:12.364 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:12.343+0000 1225d640 -1 osd.2 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:12.573 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=58}) v3 -- 0x7f24b0086d30 con 0x7f24ac0030b0 2023-08-22T23:24:12.574 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:12.574 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.839+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:12.574 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:12.574 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:12.574 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:10.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:12.574 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 23 ==== osd_map(58..58 src has 1..58) v4 ==== 1052+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:12.575 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(58..58 src has 1..58) v4 2023-08-22T23:24:12.575 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [58,58] > 57 2023-08-22T23:24:12.575 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 58 2023-08-22T23:24:12.575 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:12.576 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 58 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:12.576 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:12.577 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:12.577 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.317+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:12.577 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:11.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:24:12.579 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:12.580 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=59}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:24:12.580 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.057+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2494003970 con 0x5636bfb4b7e0 2023-08-22T23:24:12.580 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.571+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 24 ==== osd_map(59..59 src has 1..59) v4 ==== 1401+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:12.580 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.571+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(59..59 src has 1..59) v4 2023-08-22T23:24:12.580 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [59,59] > 58 2023-08-22T23:24:12.581 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 59 2023-08-22T23:24:12.581 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:13.485 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:13.457+0000 1225d640 -1 osd.2 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:14.537 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:14.525+0000 1225d640 -1 osd.2 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:15.675 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:15.578+0000 1225d640 -1 osd.2 59 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:15.912 INFO:tasks.workunit.client.0.smithi029.stdout: api api_pool: Running main() from gmock_main.cc 2023-08-22T23:24:15.912 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-08-22T23:24:15.912 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [----------] Global test environment set-up. 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5541 ms) 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4807 ms) 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3921 ms) 2023-08-22T23:24:15.913 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4470 ms) 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5035 ms) 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6468 ms) 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-08-22T23:24:15.914 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5302 ms) 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4901 ms) 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8431 ms) 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [----------] 9 tests from LibRadosPools (48876 ms total) 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [----------] Global test environment tear-down 2023-08-22T23:24:15.915 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (48876 ms total) 2023-08-22T23:24:15.916 INFO:tasks.workunit.client.0.smithi029.stdout: api_pool: [ PASSED ] 9 tests. 2023-08-22T23:24:17.023 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:17.016+0000 1225d640 -1 osd.2 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:17.084 INFO:tasks.workunit.client.0.smithi029.stdout:_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 59 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.572+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:12.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:17.085 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:13.981+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:17.086 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:13.981+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:17.086 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:13.981+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:17.086 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:14.982+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:17.086 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:14.982+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:17.086 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:14.982+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:17.086 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:15.982+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:17.087 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:15.982+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:17.087 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:15.982+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:17.087 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:16.982+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:17.087 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:16.982+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:17.087 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:16.982+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:17.087 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:17.088 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:17.088 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:17.088 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:24:17.088 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:17.088 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=60}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:24:17.088 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.058+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f2494003970 con 0x5636bfb4b7e0 2023-08-22T23:24:17.089 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 25 ==== osd_map(60..63 src has 1..63) v4 ==== 5049+0+0 (secure 0 0 0) 0x7f24b8060180 con 0x7f24ac0030b0 2023-08-22T23:24:17.089 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(60..63 src has 1..63) v4 2023-08-22T23:24:17.089 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [60,63] > 59 2023-08-22T23:24:17.089 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 60 2023-08-22T23:24:17.089 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:18.100 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:18.086+0000 1225d640 -1 osd.2 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 9.18 9:1fff0678:test-rados-api-smithi029-92451-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-08-22T23:24:18.631 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:18.577+0000 1225d640 -1 osd.6 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:19.396 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:19.353+0000 1225d640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:19.632 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:19.593+0000 1225d640 -1 osd.6 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:20.445 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:20.417+0000 1225d640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:21.215 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:21.062+0000 1225d640 -1 osd.6 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:21.604 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:21.599+0000 1225d640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:22.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 60 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 61 2023-08-22T23:24:22.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 61 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.082+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 62 2023-08-22T23:24:22.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 62 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 63 2023-08-22T23:24:22.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 63 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:22.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.083+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:22.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.982+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:22.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.982+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:22.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:17.982+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:18.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:18.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:18.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:19.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:19.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:19.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:22.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:19.983+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:24:22.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:20.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:22.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:20.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:22.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:20.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:22.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:21.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:22.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:21.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:22.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:21.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:22.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:22.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:22.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=64}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.059+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f24940039d0 con 0x5636bfb4b7e0 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.062+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 26 ==== osd_map(64..68 src has 1..68) v4 ==== 2065+0+0 (secure 0 0 0) 0x7f24b80580d0 con 0x7f24ac0030b0 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.062+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(64..68 src has 1..68) v4 2023-08-22T23:24:22.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [64,68] > 63 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 64 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 64 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 65 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 65 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 66 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 66 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 67 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 67 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 68 2023-08-22T23:24:22.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:22.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 68 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:22.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:22.454 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:22.357+0000 1225d640 -1 osd.6 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:22.766 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:22.754+0000 1225d640 -1 osd.2 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:24.024 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:23.875+0000 1225d640 -1 osd.2 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:24.050 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:24.043+0000 1225d640 -1 osd.6 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:25.018 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:24.976+0000 1225d640 -1 osd.2 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:25.229 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:25.212+0000 1225d640 -1 osd.6 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:26.399 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:26.282+0000 1225d640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:26.857 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:26.664+0000 1225d640 -1 osd.6 70 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:27.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:27.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.063+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:27.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:27.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:27.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:22.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:27.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:23.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:27.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:23.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:23.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:24.983+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:24.983+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:24.983+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:25.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:25.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:25.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:27.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:26.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:26.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:26.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:27.107 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=69}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.060+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f24940039d0 con 0x5636bfb4b7e0 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 27 ==== osd_map(69..73 src has 1..73) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f24b80580d0 con 0x7f24ac0030b0 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(69..73 src has 1..73) v4 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [69,73] > 68 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 69 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:27.108 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 69 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:27.109 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 70 2023-08-22T23:24:27.404 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:27.380+0000 1225d640 -1 osd.2 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:27.897 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08- api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-08-22T23:24:27.897 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-08-22T23:24:27.898 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-08-22T23:24:27.898 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-08-22T23:24:27.898 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (60747 ms) 2023-08-22T23:24:27.898 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (60747 ms total) 2023-08-22T23:24:27.899 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: 2023-08-22T23:24:27.899 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-08-22T23:24:27.899 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (60747 ms total) 2023-08-22T23:24:27.899 INFO:tasks.workunit.client.0.smithi029.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-08-22T23:24:28.305 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:28.303+0000 1225d640 -1 osd.6 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:28.537 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:28.449+0000 1225d640 -1 osd.2 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:29.311 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:24:29.304+0000 1225d640 -1 osd.6 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4416.0:2 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) 2023-08-22T23:24:29.489 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:29.441+0000 1225d640 -1 osd.2 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:30.623 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:30.575+0000 1225d640 -1 osd.2 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:31.651 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:24:31.619+0000 1225d640 -1 osd.2 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:2 21.7 21:e6fa9331:test-rados-api-smithi029-92698-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-08-22T23:24:31.811 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:17.095 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.822 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:17.105 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.879 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:17.175 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.882 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:17.179 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.908 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:17.191 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.917 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:17.200 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.961 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:17.245 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:31.974 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:17.271 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:32.042 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:17.325 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:32.061 INFO:tasks.workunit.client.0.smithi029.stdout:22T23:24:27.103+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 70 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.103+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 71 2023-08-22T23:24:32.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 71 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 72 2023-08-22T23:24:32.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 72 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.063 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 73 2023-08-22T23:24:32.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 73 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.064 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.104+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:27.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:28.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:32.065 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:28.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:28.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:29.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:29.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:29.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:29.984+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:30.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:30.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:32.066 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:30.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:32.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:31.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:32.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:31.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:32.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:31.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:32.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 2 client.4416.objecter tid 2 on osd.6 is laggy 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 10 monclient: _renew_subs 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=74}) v3 -- 0x7f2494002f90 con 0x7f24ac0030b0 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.061+0000 7f24c52b4640 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- ping magic: 0 v1 -- 0x7f24940039d0 con 0x5636bfb4b7e0 2023-08-22T23:24:32.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.073+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== mon.0 v2:172.21.15.29:3300/0 28 ==== osd_map(74..78 src has 1..78) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f24b80580d0 con 0x7f24ac0030b0 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.073+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(74..78 src has 1..78) v4 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [74,78] > 73 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 74 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 74 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 75 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 75 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 76 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 76 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.077 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.078 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 77 2023-08-22T23:24:32.078 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.078 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 77 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:32.078 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:32.078 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 78 2023-08-22T23:24:32.079 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 10 client.4416.objecter checking op 2 2023-08-22T23:24:32.079 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 78 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:33.610 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:33.611 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:33.611 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.074+0000 7f24c084b640 20 client.4416.objecter 2 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:33.611 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:33.611 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:33.611 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:32.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:33.611 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 2 ==== osd_op_reply(2 exist [writefull 0~5] v42'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(2 exist [writefull 0~5] v42'1 uv1 ondisk = 0) v8 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 2 ondisk uv 1 in 3.1 attempt 0 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 2 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 15 client.4416.objecter _finish_op 2 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:33.612 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 2 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 librados: Objecter returned from writefull r=0 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 78 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:33.613 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [read 0~256] tid 3 osd.6 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 3 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 15 client.4416.objecter _send_op 3 to 3.1 on osd.6 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:3 3.1 3:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78) v8 -- 0x5636bfb535d0 con 0x5636bfb4b7e0 2023-08-22T23:24:33.614 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:39.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:39.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb53930 2023-08-22T23:24:39.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 78 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:39.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:39.986 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.a7bbaa1 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:39.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:39.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 10 client.4416.objecter _op_submit oid noexist '@3' '@3' [read 0~256] tid 4 osd.6 2023-08-22T23:24:39.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:39.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 4 2023-08-22T23:24:39.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 15 client.4416.objecter _send_op 4 to 3.1 on osd.6 2023-08-22T23:24:39.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:4 3.1 3:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78) v8 -- 0x5636bfb53dd0 con 0x5636bfb4b7e0 2023-08-22T23:24:39.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:39.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.607+0000 7f24c554a780 5 client.4416.objecter 2 in flight 2023-08-22T23:24:39.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:33.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:34.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:34.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:34.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:35.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:35.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:35.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:39.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:36.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:36.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:36.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:37.061+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:37.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:37.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:37.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:39.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:38.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:38.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:39.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:38.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:39.992 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:39.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:39.992 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:39.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:42.801 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:39.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:42.802 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:39.984+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:24:42.802 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:40.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:42.802 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:40.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:42.802 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:40.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:42.802 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:41.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:41.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:41.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.062+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (9194 ms) 2023-08-22T23:24:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:42.804 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-08-22T23:24:42.804 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 3 ondisk uv 1 in 3.1 attempt 0 2023-08-22T23:24:42.804 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 5 2023-08-22T23:24:42.804 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 3 2023-08-22T23:24:42.804 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 15 client.4416.objecter _finish_op 3 2023-08-22T23:24:42.805 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:42.805 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 3 2023-08-22T23:24:42.805 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 5 client.4416.objecter 1 in flight 2023-08-22T23:24:42.806 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:42.806 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-08-22T23:24:42.806 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:42.806 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 4 ondisk uv 0 in 3.1 attempt 0 2023-08-22T23:24:42.807 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-08-22T23:24:42.807 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:42.807 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 4 2023-08-22T23:24:42.807 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 15 client.4416.objecter _finish_op 4 2023-08-22T23:24:42.808 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:42.808 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 4 2023-08-22T23:24:42.808 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:42.809 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 3 ==== osd_map(79..80 src has 1..81) v4 ==== 930+0+0 (crc 0 0 0) 0x7f24b40571e0 con 0x5636bfb4b7e0 2023-08-22T23:24:42.809 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(79..80 src has 1..81) v4 2023-08-22T23:24:42.809 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [79,80] > 78 2023-08-22T23:24:42.809 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 79 2023-08-22T23:24:42.810 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 80 2023-08-22T23:24:42.810 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:42.810 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb53930 2023-08-22T23:24:42.810 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 80 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:42.810 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:42.811 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:42.811 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.799+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:42.811 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [read 0~256] tid 5 osd.6 2023-08-22T23:24:42.811 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:42.811 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 5 2023-08-22T23:24:42.812 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 15 client.4416.objecter _send_op 5 to 3.1 on osd.6 2023-08-22T23:24:42.812 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:5 3.1 3:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80) v8 -- 0x5636bfb54130 con 0x5636bfb4b7e0 2023-08-22T23:24:42.812 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:42.812 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:42.812 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:42.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 80 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:42.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:42.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.a7bbaa1 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:42.813 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:42.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 10 client.4416.objecter _op_submit oid noexist '@3' '@3' [read 0~256] tid 6 osd.6 2023-08-22T23:24:42.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:42.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 6 2023-08-22T23:24:42.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 15 client.4416.objecter _send_op 6 to 3.1 on osd.6 2023-08-22T23:24:50.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:6 3.1 3:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80) v8 -- 0x5636bfbf9100 con 0x5636bfb4b7e0 2023-08-22T23:24:50.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:50.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.800+0000 7f24c554a780 5 client.4416.objecter 2 in flight 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:42.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:43.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:43.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:43.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:44.984+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:44.984+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:44.984+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:45.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:45.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:45.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:46.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:46.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:46.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:47.063+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:50.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:47.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:47.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:47.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:48.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:48.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:48.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:49.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:49.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:49.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:50.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:49.985+0000 7f24ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 6 ==== osd_map(81..93 src has 1..93) v4 ==== 13391+0+0 (crc 0 0 0) 0x7f24b40571e0 con 0x5636bfb4b7e0 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(81..93 src has 1..93) v4 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [81,93] > 80 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 81 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 81 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 81 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 82 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 82 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 82 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 83 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 83 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 83 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 84 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 84 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 84 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 85 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 85 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.537 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 85 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 86 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 86 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 86 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.538 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 87 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 87 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 87 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.539 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 88 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 88 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 88 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.540 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 89 2023-08-22T23:24:50.541 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.541 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 89 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.541 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.541 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 89 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 90 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 90 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.542 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 90 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 91 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 91 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.543 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 91 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 92 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 92 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.544 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 92 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 93 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 5 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 93 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 10 client.4416.objecter checking op 6 2023-08-22T23:24:50.545 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 93 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter 5 3.d3a59201 osd.6 exist [read 0~256] 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.526+0000 7f24c084b640 20 client.4416.objecter 6 3.a7bbaa1 osd.6 noexist [read 0~256] 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:50.546 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 5 ondisk uv 1 in 3.1 attempt 0 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 5 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 5 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 15 client.4416.objecter _finish_op 5 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 5 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 5 client.4416.objecter 1 in flight 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:50.547 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-08-22T23:24:50.548 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:50.548 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 6 ondisk uv 0 in 3.1 attempt 0 2023-08-22T23:24:50.549 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-08-22T23:24:50.549 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:50.549 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 6 2023-08-22T23:24:50.549 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 15 client.4416.objecter _finish_op 6 2023-08-22T23:24:50.549 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:50.549 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 6 2023-08-22T23:24:50.550 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:50.550 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (7727 ms) 2023-08-22T23:24:50.550 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-08-22T23:24:50.550 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb53930 2023-08-22T23:24:50.550 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 93 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:50.550 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:51.977 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:37.260 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:51.987 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:37.270 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:51.991 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:37.288 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:51.994 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:37.291 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:51.995 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:37.279 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:51.999 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:37.282 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:52.158 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:37.455 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:52.169 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:03:37.453 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:52.213 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:03:37.496 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:24:52.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:52.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:52.987 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [read 0~256] tid 7 osd.6 2023-08-22T23:24:52.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:52.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 7 2023-08-22T23:24:52.988 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 15 client.4416.objecter _send_op 7 to 3.1 on osd.6 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:7 3.1 3:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5636bfc19470 con 0x5636bfb4b7e0 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfc397e0 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 93 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:52.989 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.a7bbaa1 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 10 client.4416.objecter _op_submit oid noexist '@3' '@3' [read 0~256] tid 8 osd.6 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 8 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 15 client.4416.objecter _send_op 8 to 3.1 on osd.6 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:8 3.1 3:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5636bfc39c80 con 0x5636bfb4b7e0 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:52.990 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.527+0000 7f24c554a780 5 client.4416.objecter 2 in flight 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:50.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:51.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:51.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:51.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:52.991 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:52.064+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:52.992 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:52.985+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:52.992 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:52.985+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:53.435 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:52.985+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:53.435 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 9 ==== osd_map(94..96 src has 1..96) v4 ==== 5468+0+0 (crc 0 0 0) 0x7f24b40571e0 con 0x5636bfb4b7e0 2023-08-22T23:24:53.435 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(94..96 src has 1..96) v4 2023-08-22T23:24:53.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [94,96] > 93 2023-08-22T23:24:53.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 94 2023-08-22T23:24:53.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter checking op 7 2023-08-22T23:24:53.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 94 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:53.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:53.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter checking op 8 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 94 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 95 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter checking op 7 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 95 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:53.437 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter checking op 8 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 95 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 96 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter checking op 7 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 96 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 10 client.4416.objecter checking op 8 2023-08-22T23:24:53.438 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 96 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter 7 3.d3a59201 osd.6 exist [read 0~256] 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.415+0000 7f24c084b640 20 client.4416.objecter 8 3.a7bbaa1 osd.6 noexist [read 0~256] 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2906 ms) 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 10 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:53.439 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 7 ondisk uv 1 in 3.1 attempt 0 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 5 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 7 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 15 client.4416.objecter _finish_op 7 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:53.440 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 7 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 5 client.4416.objecter 1 in flight 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 11 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 8 ondisk uv 0 in 3.1 attempt 0 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:53.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 8 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 15 client.4416.objecter _finish_op 8 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 8 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 librados: queue_aio_write 0x5636bfa3ab50 completion 0x5636bfb53930 write_seq 1 2023-08-22T23:24:53.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:53.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 96 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:53.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:53.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:53.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:53.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [write 0~5 in=5b] tid 9 osd.6 2023-08-22T23:24:56.954 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:56.955 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 9 2023-08-22T23:24:56.955 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 15 client.4416.objecter _send_op 9 to 3.1 on osd.6 2023-08-22T23:24:56.955 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:9 3.1 3:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5636bfb53a40 con 0x5636bfb4b7e0 2023-08-22T23:24:56.955 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.433+0000 7f24c554a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.988+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.988+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:53.988+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:54.989+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:56.957 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:54.989+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:54.989+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 12 ==== osd_map(97..98 src has 1..98) v4 ==== 2019+0+0 (crc 0 0 0) 0x7f24b40571e0 con 0x5636bfb4b7e0 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(97..98 src has 1..98) v4 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [97,98] > 96 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 97 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 10 client.4416.objecter checking op 9 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 97 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:56.958 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 98 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 10 client.4416.objecter checking op 9 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 98 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.678+0000 7f24c084b640 20 client.4416.objecter 9 3.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.989+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:56.959 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.989+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:56.960 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:55.989+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:56.960 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (3519 ms) 2023-08-22T23:24:56.960 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-08-22T23:24:57.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 13 ==== osd_op_reply(9 exist [write 0~5] v98'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:57.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(9 exist [write 0~5] v98'2 uv2 ondisk = 0) v8 2023-08-22T23:24:57.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:57.067 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 9 ondisk uv 2 in 3.1 attempt 0 2023-08-22T23:24:57.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:57.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 9 2023-08-22T23:24:57.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 15 client.4416.objecter _finish_op 9 2023-08-22T23:24:57.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:57.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 9 2023-08-22T23:24:57.068 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:57.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c284f640 20 librados: complete_aio_write 0x5636bfb53930 2023-08-22T23:24:57.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-08-22T23:24:57.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 20 librados: queue_aio_write 0x5636bfa3ab50 completion 0x5636bfb560e0 write_seq 2 2023-08-22T23:24:57.069 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 98 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.951+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [write 0~5 in=5b] tid 10 osd.6 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:57.070 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 10 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 15 client.4416.objecter _send_op 10 to 3.1 on osd.6 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:10 3.1 3:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v8 -- 0x5636bfb53a40 con 0x5636bfb4b7e0 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.952+0000 7f24c554a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.989+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.989+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:57.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:56.989+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:58.033 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.065+0000 7f24c52b4640 10 client.4416.objecter tick 2023-08-22T23:24:58.033 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 14 ==== osd_map(99..100 src has 1..100) v4 ==== 5349+0+0 (crc 0 0 0) 0x7f24b40571e0 con 0x5636bfb4b7e0 2023-08-22T23:24:58.033 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(99..100 src has 1..100) v4 2023-08-22T23:24:58.033 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [99,100] > 98 2023-08-22T23:24:58.033 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 99 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 10 client.4416.objecter checking op 10 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 99 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 100 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 10 client.4416.objecter checking op 10 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 100 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:58.034 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:58.035 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:58.035 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.811+0000 7f24c084b640 20 client.4416.objecter 10 3.d3a59201 osd.6 exist [write 0~5 in=5b] 2023-08-22T23:24:58.035 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.989+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:58.036 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.989+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:58.036 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:57.989+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:58.036 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 15 ==== osd_op_reply(10 exist [write 0~5] v100'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.037 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(10 exist [write 0~5] v100'3 uv3 ondisk = 0) v8 2023-08-22T23:24:58.037 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.038 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 10 ondisk uv 3 in 3.1 attempt 0 2023-08-22T23:24:58.038 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:58.039 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 10 2023-08-22T23:24:58.039 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 15 client.4416.objecter _finish_op 10 2023-08-22T23:24:58.039 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.041 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 10 2023-08-22T23:24:58.041 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:58.041 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c284f640 20 librados: complete_aio_write 0x5636bfb560e0 2023-08-22T23:24:58.041 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1079 ms) 2023-08-22T23:24:58.041 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-08-22T23:24:58.041 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-08-22T23:24:58.239 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 librados: queue_aio_write 0x5636bfa3ab50 completion 0x5636bfb53a40 write_seq 3 2023-08-22T23:24:58.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:58.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:58.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:58.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [write 0~5 in=5b] tid 11 osd.6 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 11 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 15 client.4416.objecter _send_op 11 to 3.1 on osd.6 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:11 3.1 3:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbfce40 con 0x5636bfb4b7e0 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.031+0000 7f24c554a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-08-22T23:24:58.241 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.237+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 16 ==== osd_op_reply(11 exist [write 0~5] v100'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.237+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(11 exist [write 0~5] v100'4 uv4 ondisk = 0) v8 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 11 ondisk uv 4 in 3.1 attempt 0 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 11 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 15 client.4416.objecter _finish_op 11 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.242 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 11 2023-08-22T23:24:58.243 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:58.243 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c284f640 20 librados: complete_aio_write 0x5636bfb53a40 2023-08-22T23:24:58.243 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (207 ms) 2023-08-22T23:24:58.243 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-08-22T23:24:58.243 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:58.243 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:58.320 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:58.321 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.321 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.321 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [read 0~0] tid 12 osd.6 2023-08-22T23:24:58.321 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.321 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 12 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 15 client.4416.objecter _send_op 12 to 3.1 on osd.6 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:12 3.1 3:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbfc7d0 con 0x5636bfb4b7e0 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfbfcc40 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:58.322 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.a7bbaa1 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 10 client.4416.objecter _op_submit oid noexist '@3' '@3' [read 0~0] tid 13 osd.6 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 13 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 15 client.4416.objecter _send_op 13 to 3.1 on osd.6 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:13 3.1 3:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbf9460 con 0x5636bfb4b7e0 2023-08-22T23:24:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:58.324 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.238+0000 7f24c554a780 5 client.4416.objecter 2 in flight 2023-08-22T23:24:58.324 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 17 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.324 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-08-22T23:24:58.324 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.324 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 12 ondisk uv 4 in 3.1 attempt 0 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 5 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 12 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 15 client.4416.objecter _finish_op 12 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 12 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.325 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 18 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 13 ondisk uv 0 in 3.1 attempt 0 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 13 2023-08-22T23:24:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 15 client.4416.objecter _finish_op 13 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 13 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (81 ms) 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:58.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [read 0~0] tid 14 osd.6 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 14 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 15 client.4416.objecter _send_op 14 to 3.1 on osd.6 2023-08-22T23:24:58.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:14 3.1 3:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbfc7d0 con 0x5636bfb4b7e0 2023-08-22T23:24:58.379 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfbfcd50 2023-08-22T23:24:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.a7bbaa1 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 10 client.4416.objecter _op_submit oid noexist '@3' '@3' [read 0~0] tid 15 osd.6 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 15 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 15 client.4416.objecter _send_op 15 to 3.1 on osd.6 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:15 3.1 3:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbf9460 con 0x5636bfb4b7e0 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.319+0000 7f24c554a780 5 client.4416.objecter 2 in flight 2023-08-22T23:24:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 14 ondisk uv 4 in 3.1 attempt 0 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 5 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 14 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 15 client.4416.objecter _finish_op 14 2023-08-22T23:24:58.382 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 14 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.377+0000 7f24c284f640 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 15 ondisk uv 0 in 3.1 attempt 0 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 10 client.4416.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-08-22T23:24:58.383 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 15 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 15 client.4416.objecter _finish_op 15 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 15 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (59 ms) 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-08-22T23:24:58.384 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:58.385 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base exist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:58.385 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:58.385 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.385 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.385 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [read 0~0] tid 16 osd.6 2023-08-22T23:24:58.385 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:58.386 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 16 2023-08-22T23:24:58.386 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 15 client.4416.objecter _send_op 16 to 3.1 on osd.6 2023-08-22T23:24:58.386 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:16 3.1 3:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbf9460 con 0x5636bfb4b7e0 2023-08-22T23:24:58.386 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.386 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfbf9e80 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base noexist @3 precalc_pgid 0 pgid 0.0 is_read 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter _calc_target target noexist @3 -> pgid 3.a7bbaa1 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.a7bbaa1 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 10 client.4416.objecter _op_submit oid noexist '@3' '@3' [read 0~0] tid 17 osd.6 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.387 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 17 2023-08-22T23:24:58.441 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 15 client.4416.objecter _send_op 17 to 3.1 on osd.6 2023-08-22T23:24:58.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:17 3.1 3:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbfcc40 con 0x5636bfb4b7e0 2023-08-22T23:24:58.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:58.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.378+0000 7f24c554a780 5 client.4416.objecter 2 in flight 2023-08-22T23:24:58.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 21 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-08-22T23:24:58.442 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 16 ondisk uv 4 in 3.1 attempt 0 2023-08-22T23:24:58.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 5 2023-08-22T23:24:58.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 16 2023-08-22T23:24:58.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 15 client.4416.objecter _finish_op 16 2023-08-22T23:24:58.443 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 5 2023-08-22T23:24:58.444 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 16 2023-08-22T23:24:58.444 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 5 client.4416.objecter 1 in flight 2023-08-22T23:24:58.444 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 22 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:58.444 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-08-22T23:24:58.445 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:58.445 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 17 ondisk uv 0 in 3.1 attempt 0 2023-08-22T23:24:58.445 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-08-22T23:24:58.445 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:58.445 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 17 2023-08-22T23:24:58.446 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 15 client.4416.objecter _finish_op 17 2023-08-22T23:24:58.446 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:58.446 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 17 2023-08-22T23:24:58.446 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.439+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:58.446 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (62 ms) 2023-08-22T23:24:58.447 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-08-22T23:24:58.447 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 20 librados: queue_aio_write 0x5636bfa3ab50 completion 0x5636bfbfc8e0 write_seq 4 2023-08-22T23:24:58.447 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:59.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 100 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:59.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:59.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:59.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:59.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [writefull 0~5 in=5b] tid 18 osd.6 2023-08-22T23:24:59.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:59.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 18 2023-08-22T23:24:59.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 15 client.4416.objecter _send_op 18 to 3.1 on osd.6 2023-08-22T23:24:59.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:18 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) v8 -- 0x5636bfbf9460 con 0x5636bfb4b7e0 2023-08-22T23:24:59.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:59.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.440+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:59.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.990+0000 7f24ab7fe640 10 monclient: tick 2023-08-22T23:24:59.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.990+0000 7f24ab7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:24:59.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:58.990+0000 7f24ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:24:59.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 23 ==== osd_map(101..101 src has 1..101) v4 ==== 709+0+0 (crc 0 0 0) 0x7f24b40571e0 con 0x5636bfb4b7e0 2023-08-22T23:24:59.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_map(101..101 src has 1..101) v4 2023-08-22T23:24:59.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 3 client.4416.objecter handle_osd_map got epochs [101,101] > 100 2023-08-22T23:24:59.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 3 client.4416.objecter handle_osd_map decoding incremental epoch 101 2023-08-22T23:24:59.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 10 client.4416.objecter checking op 18 2023-08-22T23:24:59.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 20 client.4416.objecter _calc_target epoch 101 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:59.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:59.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 20 client.4416.objecter dump_active .. 0 homeless 2023-08-22T23:24:59.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.019+0000 7f24c084b640 20 client.4416.objecter 18 3.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2023-08-22T23:24:59.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 24 ==== osd_op_reply(18 exist [writefull 0~5] v101'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:59.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(18 exist [writefull 0~5] v101'5 uv5 ondisk = 0) v8 2023-08-22T23:24:59.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:59.232 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 18 ondisk uv 5 in 3.1 attempt 0 2023-08-22T23:24:59.233 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:59.233 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 18 2023-08-22T23:24:59.233 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 15 client.4416.objecter _finish_op 18 2023-08-22T23:24:59.234 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:59.234 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 18 2023-08-22T23:24:59.234 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:59.234 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.184+0000 7f24c284f640 20 librados: complete_aio_write 0x5636bfbfc8e0 2023-08-22T23:24:59.235 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (745 ms) 2023-08-22T23:24:59.235 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-08-22T23:24:59.235 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 20 librados: queue_aio_write 0x5636bfa3ab50 completion 0x5636bfbfd020 write_seq 5 2023-08-22T23:24:59.235 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:59.236 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 101 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:59.236 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:59.236 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:59.236 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:59.237 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [writefull 0~5 in=5b] tid 19 osd.6 2023-08-22T23:24:59.237 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:59.237 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 19 2023-08-22T23:24:59.237 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 15 client.4416.objecter _send_op 19 to 3.1 on osd.6 2023-08-22T23:24:59.238 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:19 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) v8 -- 0x5636bfbf9460 con 0x5636bfb4b7e0 2023-08-22T23:24:59.238 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:59.238 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.185+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:59.238 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 25 ==== osd_op_reply(19 exist [writefull 0~5] v101'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:59.239 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(19 exist [writefull 0~5] v101'6 uv6 ondisk = 0) v8 2023-08-22T23:24:59.239 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:59.239 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 19 ondisk uv 6 in 3.1 attempt 0 2023-08-22T23:24:59.239 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:59.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 19 2023-08-22T23:24:59.240 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 15 client.4416.objecter _finish_op 19 2023-08-22T23:24:59.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:59.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 19 2023-08-22T23:24:59.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:59.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c284f640 20 librados: complete_aio_write 0x5636bfbfd020 2023-08-22T23:24:59.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (46 ms) 2023-08-22T23:24:59.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-08-22T23:24:59.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 20 librados: queue_aio_write 0x5636bfa3ab50 completion 0x5636bfbfd130 write_seq 6 2023-08-22T23:24:59.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 10 client.4416.objecter _op_submit op 0x5636bfb55c40 2023-08-22T23:24:59.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 20 client.4416.objecter _calc_target epoch 101 base exist @3 precalc_pgid 0 pgid 0.0 is_write 2023-08-22T23:24:59.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 20 client.4416.objecter _calc_target target exist @3 -> pgid 3.d3a59201 2023-08-22T23:24:59.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 10 client.4416.objecter _calc_target raw pgid 3.d3a59201 -> actual 3.1 acting [6,2] primary 6 2023-08-22T23:24:59.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 20 client.4416.objecter _get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:59.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 10 client.4416.objecter _op_submit oid exist '@3' '@3' [writefull 0~5 in=5b] tid 20 osd.6 2023-08-22T23:24:59.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 20 client.4416.objecter get_session s=0x5636bf9fc050 osd=6 3 2023-08-22T23:24:59.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 15 client.4416.objecter _session_op_assign 6 20 2023-08-22T23:24:59.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 15 client.4416.objecter _send_op 20 to 3.1 on osd.6 2023-08-22T23:24:59.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- osd_op(unknown.0.0:20 3.1 3:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) v8 -- 0x5636bfbf9460 con 0x5636bfb4b7e0 2023-08-22T23:24:59.284 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:59.284 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.231+0000 7f24c554a780 5 client.4416.objecter 1 in flight 2023-08-22T23:24:59.284 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 1 -- 172.21.15.29:0/534600864 <== osd.6 v2:172.21.15.52:6818/2556931765 26 ==== osd_op_reply(20 exist [writefull 0~5] v101'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f24b405f040 con 0x5636bfb4b7e0 2023-08-22T23:24:59.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 10 client.4416.objecter ms_dispatch 0x5636bfa14300 osd_op_reply(20 exist [writefull 0~5] v101'7 uv7 ondisk = 0) v8 2023-08-22T23:24:59.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 10 client.4416.objecter in handle_osd_op_reply 2023-08-22T23:24:59.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 7 client.4416.objecter handle_osd_op_reply 20 ondisk uv 7 in 3.1 attempt 0 2023-08-22T23:24:59.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 10 client.4416.objecter op 0 rval 0 len 0 2023-08-22T23:24:59.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 15 client.4416.objecter handle_osd_op_reply completed tid 20 2023-08-22T23:24:59.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 15 client.4416.objecter _finish_op 20 2023-08-22T23:24:59.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 4 2023-08-22T23:24:59.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 15 client.4416.objecter _session_op_remove 6 20 2023-08-22T23:24:59.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 5 client.4416.objecter 0 in flight 2023-08-22T23:24:59.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c284f640 20 librados: complete_aio_write 0x5636bfbfd130 2023-08-22T23:24:59.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (47 ms) 2023-08-22T23:24:59.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.277+0000 7f24c554a780 10 librados: watch_flush enter 2023-08-22T23:24:59.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 10 librados: watch_flush exit 2023-08-22T23:24:59.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 10 client.4416.objecter close_session for osd.6 2023-08-22T23:24:59.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x5636bfb4b7e0 msgr2=0x5636bfb4bbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:24:59.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x5636bfb4b7e0 0x5636bfb4bbc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:24:59.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 20 client.4416.objecter put_session s=0x5636bf9fc050 osd=6 1 2023-08-22T23:24:59.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 20 client.4416.objecter shutdown clearing up homeless session... 2023-08-22T23:24:59.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 10 client.4416.objecter successfully canceled tick 2023-08-22T23:24:59.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f24b0069c20 msgr2=0x7f24b006c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:24:59.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f24b0069c20 0x7f24b006c0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f24bc002230 tx=0x7f24bc008000 comp rx=0 tx=0).stop 2023-08-22T23:24:59.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 10 monclient: shutdown 2023-08-22T23:24:59.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:24:59.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24ac0030b0 msgr2=0x5636bfbefd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:24:59.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 --2- 172.21.15.29:0/534600864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24ac0030b0 0x5636bfbefd90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x5636bfac8040 tx=0x7f24b8004d10 comp rx=0 tx=0).stop 2023-08-22T23:24:59.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c3851640 1 -- 172.21.15.29:0/534600864 reap_dead start 2023-08-22T23:24:59.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 shutdown_connections 2023-08-22T23:24:59.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 >> 172.21.15.29:0/534600864 conn(0x5636bfacbba0 msgr2=0x5636bfb4dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:24:59.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 shutdown_connections 2023-08-22T23:24:59.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 -- 172.21.15.29:0/534600864 wait complete. 2023-08-22T23:24:59.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.278+0000 7f24c554a780 1 librados: shutdown 2023-08-22T23:24:59.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [----------] 12 tests from AsioRados (25672 ms total) 2023-08-22T23:24:59.293 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: 2023-08-22T23:24:59.294 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [----------] Global test environment tear-down 2023-08-22T23:24:59.294 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (92328 ms total) 2023-08-22T23:24:59.294 INFO:tasks.workunit.client.0.smithi029.stdout: api_asio: [ PASSED ] 12 tests. 2023-08-22T23:25:01.997 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-08-22T23:25:01.998 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-08-22T23:25:01.998 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-08-22T23:25:01.998 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-08-22T23:25:01.998 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-08-22T23:25:02.001 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify 2023-08-22T23:25:02.002 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (28389 ms) 2023-08-22T23:25:02.002 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-08-22T23:25:02.002 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (12170 ms) 2023-08-22T23:25:02.003 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (40559 ms total) 2023-08-22T23:25:02.003 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: 2023-08-22T23:25:02.003 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-08-22T23:25:02.003 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-08-22T23:25:02.004 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify 2023-08-22T23:25:02.004 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (23691 ms) 2023-08-22T23:25:02.004 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-08-22T23:25:02.004 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify 2023-08-22T23:25:02.005 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (16784 ms) 2023-08-22T23:25:02.005 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-08-22T23:25:02.005 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (1147 ms) 2023-08-22T23:25:02.006 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-08-22T23:25:02.006 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (243 ms) 2023-08-22T23:25:02.006 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-08-22T23:25:02.007 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775577934160 notify_id 433791696897 notifier_gid 4723 2023-08-22T23:25:02.007 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (524 ms) 2023-08-22T23:25:02.007 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-08-22T23:25:02.007 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775577934160 notify_id 433791696897 notifier_gid 4723 2023-08-22T23:25:02.007 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775577934160 notify_id 433791696898 notifier_gid 4723 2023-08-22T23:25:02.008 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (486 ms) 2023-08-22T23:25:02.008 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-08-22T23:25:02.008 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775579583840 notify_id 433791696899 notifier_gid 4723 2023-08-22T23:25:02.008 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (215 ms) 2023-08-22T23:25:02.009 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-08-22T23:25:02.009 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578704576 notify_id 433791696900 notifier_gid 4723 2023-08-22T23:25:02.009 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (185 ms) 2023-08-22T23:25:02.009 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-08-22T23:25:02.009 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578691328 notify_id 433791696901 notifier_gid 4723 2023-08-22T23:25:02.009 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (253 ms) 2023-08-22T23:25:02.010 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-08-22T23:25:02.010 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578691328 notify_id 433791696902 notifier_gid 4723 2023-08-22T23:25:02.011 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (282 ms) 2023-08-22T23:25:02.011 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-08-22T23:25:02.011 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: trying... 2023-08-22T23:25:05.961 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578691328 notify_ api_cmd_pp: Running main() from gmock_main.cc 2023-08-22T23:25:05.961 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-08-22T23:25:05.962 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-08-22T23:25:05.962 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-08-22T23:25:05.962 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-08-22T23:25:05.962 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (628 ms) 2023-08-22T23:25:05.962 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-08-22T23:25:05.962 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (140 ms) 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (98109 ms) 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (98877 ms total) 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (98877 ms total) 2023-08-22T23:25:05.963 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-08-22T23:25:09.260 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: Running main() from gmock_main.cc 2023-08-22T23:25:09.260 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-08-22T23:25:09.260 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (24031 ms) 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-08-22T23:25:09.261 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (17519 ms) 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (7343 ms) 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (15028 ms) 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (3845 ms) 2023-08-22T23:25:09.262 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-08-22T23:25:09.263 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7012 ms) 2023-08-22T23:25:09.263 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-08-22T23:25:09.263 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5388 ms) 2023-08-22T23:25:09.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-08-22T23:25:09.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (5453 ms) 2023-08-22T23:25:09.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-08-22T23:25:09.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (275 ms) 2023-08-22T23:25:09.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-08-22T23:25:09.264 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (223 ms) 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (203 ms) 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (1298 ms) 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (1112 ms) 2023-08-22T23:25:09.265 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (768 ms) 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (340 ms) 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (122 ms) 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (89960 ms total) 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: 2023-08-22T23:25:09.266 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-08-22T23:25:09.267 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (102067 ms total) 2023-08-22T23:25:09.267 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-08-22T23:25:13.634 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-08-22T23:25:13.635 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-08-22T23:25:13.635 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-08-22T23:25:13.635 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-08-22T23:25:13.635 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: seed 92670 2023-08-22T23:25:13.636 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-08-22T23:25:13.636 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (14449 ms) 2023-08-22T23:25:13.636 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-08-22T23:25:13.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (21251 ms) 2023-08-22T23:25:13.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-08-22T23:25:13.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (15 ms) 2023-08-22T23:25:13.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-08-22T23:25:13.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (4598 ms) 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (11781 ms) 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (52094 ms total) 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-08-22T23:25:13.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (19212 ms) 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (32 ms) 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (4609 ms) 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7113 ms) 2023-08-22T23:25:13.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (30966 ms total) 2023-08-22T23:25:13.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: 2023-08-22T23:25:13.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-08-22T23:25:13.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (106603 ms total) 2023-08-22T23:25:13.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-08-22T23:25:22.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-08-22T23:25:22.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] Global test environment set-up. 2023-08-22T23:25:22.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] 7 tests from LibRadosList 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.ListObjects (88098 ms) 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (2553 ms) 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset foo1,foo2,foo3 2023-08-22T23:25:22.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo1 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo2 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo3 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset foo1,foo4,foo5 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo4 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo5 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo1 2023-08-22T23:25:22.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset foo6,foo7 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo7 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo6 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns1:foo4 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns1:foo5 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns2:foo7 2023-08-22T23:25:22.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns2:foo6 2023-08-22T23:25:22.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns1:foo1 2023-08-22T23:25:22.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: :foo1 2023-08-22T23:25:22.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: :foo2 2023-08-22T23:25:22.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: :foo3 2023-08-22T23:25:22.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (6516 ms) 2023-08-22T23:25:22.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-08-22T23:25:22.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 1 0 2023-08-22T23:25:22.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 10 0 2023-08-22T23:25:22.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 13 0 2023-08-22T23:25:22.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 7 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 14 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 0 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 15 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 11 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 5 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 8 0 2023-08-22T23:25:22.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 6 0 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 3 0 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 4 0 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 12 0 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 9 0 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 2 0 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-08-22T23:25:22.222 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (8696 ms) 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: x cursor=MIN 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=1 cursor=11:02547ec2:::1:head 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=10 cursor=11:52ea6a34:::10:head 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=13 cursor=11:566253c9:::13:head 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=7 cursor=11:5c6b0b28:::7:head 2023-08-22T23:25:22.223 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=14 cursor=11:62a1935d:::14:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=0 cursor=11:6cac518f:::0:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=15 cursor=11:863748b0:::15:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=11 cursor=11:89d3ae78:::11:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=5 cursor=11:b29083e3:::5:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=8 cursor=11:bd63b0f1:::8:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=6 cursor=11:c4fdafeb:::6:head 2023-08-22T23:25:22.224 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=3 cursor=11:cfc208b3:::3:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=4 cursor=11:d83876eb:::4:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=12 cursor=11:de5d7c5f:::12:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=9 cursor=11:e960b815:::9:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > oid=2 cursor=11:f905c69b:::2:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: FIRST> seek to MIN oid=1 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=1 cursor=11:02547ec2:::1:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:02547ec2:::1:head 2023-08-22T23:25:22.225 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:02547ec2:::1:head -> 1 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=10 cursor=11:52ea6a34:::10:head 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:52ea6a34:::10:head 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=13 cursor=11:566253c9:::13:head 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:566253c9:::13:head 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:566253c9:::13:head -> 13 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=7 cursor=11:5c6b0b28:::7:head 2023-08-22T23:25:22.226 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2023-08-22T23:25:22.227 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2023-08-22T23:25:26.336 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=14 cursor=11:62a1935d:::14:head 2023-08-22T23:25:26.336 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:62a1935d:::14:head 2023-08-22T23:25:26.337 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:62a1935d:::14:head -> 14 2023-08-22T23:25:26.337 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=0 cursor=11:6cac518f:::0:head 2023-08-22T23:25:26.337 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:6cac518f:::0:head 2023-08-22T23:25:26.337 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:6cac518f:::0:head -> 0 2023-08-22T23:25:26.338 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=15 cursor=11:863748b0:::15:head 2023-08-22T23:25:26.338 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:863748b0:::15:head 2023-08-22T23:25:26.338 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:863748b0:::15:head -> 15 2023-08-22T23:25:26.338 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=11 cursor=11:89d3ae78:::11:head 2023-08-22T23:25:26.339 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:89d3ae78:::11:head 2023-08-22T23:25:26.339 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2023-08-22T23:25:26.339 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=5 cursor=11:b29083e3:::5:head 2023-08-22T23:25:26.339 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:b29083e3:::5:head 2023-08-22T23:25:26.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:b29083e3:::5:head -> 5 2023-08-22T23:25:26.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=8 cursor=11:bd63b0f1:::8:head 2023-08-22T23:25:26.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2023-08-22T23:25:26.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2023-08-22T23:25:26.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=6 cursor=11:c4fdafeb:::6:head 2023-08-22T23:25:26.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2023-08-22T23:25:26.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2023-08-22T23:25:26.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=3 cursor=11:cfc208b3:::3:head 2023-08-22T23:25:26.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:cfc208b3:::3:head 2023-08-22T23:25:26.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2023-08-22T23:25:26.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=4 cursor=11:d83876eb:::4:head 2023-08-22T23:25:26.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:d83876eb:::4:head 2023-08-22T23:25:26.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:d83876eb:::4:head -> 4 2023-08-22T23:25:26.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=12 cursor=11:de5d7c5f:::12:head 2023-08-22T23:25:26.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2023-08-22T23:25:26.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2023-08-22T23:25:26.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=9 cursor=11:e960b815:::9:head 2023-08-22T23:25:26.345 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:e960b815:::9:head 2023-08-22T23:25:26.345 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:e960b815:::9:head -> 9 2023-08-22T23:25:26.345 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : oid=2 cursor=11:f905c69b:::2:head 2023-08-22T23:25:26.345 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:f905c69b:::2:head 2023-08-22T23:25:26.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:f905c69b:::2:head -> 2 2023-08-22T23:25:26.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:52ea6a34:::10:head 2023-08-22T23:25:26.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:52ea6a34:::10:head expected=11:52ea6a34:::10:head 2023-08-22T23:25:26.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2023-08-22T23:25:26.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=10 expected=10 2023-08-22T23:25:26.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:6cac518f:::0:head 2023-08-22T23:25:26.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:6cac518f:::0:head expected=11:6cac518f:::0:head 2023-08-22T23:25:26.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:6cac518f:::0:head -> 0 2023-08-22T23:25:26.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=0 expected=0 2023-08-22T23:25:26.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:62a1935d:::14:head 2023-08-22T23:25:26.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:62a1935d:::14:head expected=11:62a1935d:::14:head 2023-08-22T23:25:26.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:62a1935d:::14:head -> 14 2023-08-22T23:25:26.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=14 expected=14 2023-08-22T23:25:26.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2023-08-22T23:25:26.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:5c6b0b28:::7:head expected=11:5c6b0b28:::7:head 2023-08-22T23:25:26.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2023-08-22T23:25:26.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=7 expected=7 2023-08-22T23:25:26.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:f905c69b:::2:head 2023-08-22T23:25:26.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:f905c69b:::2:head expected=11:f905c69b:::2:head 2023-08-22T23:25:26.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:f905c69b:::2:head -> 2 2023-08-22T23:25:26.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=2 expected=2 2023-08-22T23:25:26.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:e960b815:::9:head 2023-08-22T23:25:26.352 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:e960b815:::9:head expected=11:e960b815:::9:head 2023-08-22T23:25:26.352 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:e960b815:::9:head -> 9 2023-08-22T23:25:26.352 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=9 expected=9 2023-08-22T23:25:26.353 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2023-08-22T23:25:26.353 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:de5d7c5f:::12:head expected=11:de5d7c5f:::12:head 2023-08-22T23:25:26.353 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2023-08-22T23:25:26.353 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=12 expected=12 2023-08-22T23:25:26.353 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2023-08-22T23:25:26.354 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:bd63b0f1:::8:head expected=11:bd63b0f1:::8:head 2023-08-22T23:25:26.354 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2023-08-22T23:25:27.884 INFO:tasks.workunit.client.0.smithi029.stdout: api_l api_cmd: Running main() from gmock_main.cc 2023-08-22T23:25:27.885 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-08-22T23:25:27.885 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [----------] Global test environment set-up. 2023-08-22T23:25:27.885 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-08-22T23:25:27.886 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-08-22T23:25:27.886 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (937 ms) 2023-08-22T23:25:27.886 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-08-22T23:25:27.886 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (77 ms) 2023-08-22T23:25:27.887 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-08-22T23:25:27.887 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (101889 ms) 2023-08-22T23:25:27.887 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-08-22T23:25:27.887 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:05.577329+0000 mon.a [WRN] Health check update: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-08-22T23:25:27.888 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:08.029641+0000 mon.c [INF] from='client.4814 172.21.15.29:0/178524939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi029-92921-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-08-22T23:25:27.888 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:08.178630+0000 mon.a [INF] from='client.? 172.21.15.29:0/1214314455' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi029-92481-4", "tierpool": "test-rados-api-smithi029-92481-4-cache"}]': finished 2023-08-22T23:25:27.888 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:08.184158+0000 mon.a [INF] from='client.? 172.21.15.29:0/3260422899' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi029-92670-7"}]': finished 2023-08-22T23:25:27.889 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:08.949134+0000 mon.a [INF] from='client.? 172.21.15.29:0/3260422899' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi029-92670-7"}]: dispatch 2023-08-22T23:25:27.889 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:09.149501+0000 mon.a [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-08-22T23:25:27.889 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:09.664803+0000 mon.a [INF] from='client.4814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi029-92921-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-08-22T23:25:27.889 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:10.672403+0000 mon.a [INF] from='client.? 172.21.15.29:0/911033316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi029-92291-16"}]: dispatch 2023-08-22T23:25:27.890 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:11.024990+0000 mon.b [INF] from='client.? 172.21.15.29:0/1502112702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi029-92309-22"}]: dispatch 2023-08-22T23:25:27.890 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:12.040796+0000 mon.a [INF] from='client.? 172.21.15.29:0/3260422899' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi029-92670-7"}]': finished 2023-08-22T23:25:27.890 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:12.267824+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi029-92309-22"}]: dispatch 2023-08-22T23:25:27.890 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:12.521827+0000 mon.c [INF] from='client.? 172.21.15.29:0/2118062857' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-08-22T23:25:27.891 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:13.053987+0000 mon.a [INF] from='client.? 172.21.15.29:0/911033316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi029-92291-16"}]: dispatch 2023-08-22T23:25:27.891 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:13.136772+0000 client.admin [INF] onexx 2023-08-22T23:25:27.891 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:13.147334+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-08-22T23:25:32.363 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:16.858027+0000 mon.b [INF] from='client.? 172.21.15.29:0/1502112702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi029-92309-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-08-22T23:25:32.363 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:18.073051+0000 mon.c [INF] from='client.? 172.21.15.29:0/2118062857' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-08-22T23:25:32.363 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:18.568522+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi029-92309-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-08-22T23:25:32.363 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:18.830851+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi029-92278-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-08-22T23:25:32.364 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:20.095079+0000 client.admin [INF] twoxx 2023-08-22T23:25:32.364 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:20.127488+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-08-22T23:25:32.364 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:21.608144+0000 mon.b [INF] from='client.? 172.21.15.29:0/861166832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi029-92365-10"}]: dispatch 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:22.171136+0000 mon.a [INF] from='client.? 172.21.15.29:0/911033316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi029-92291-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:22.176882+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi029-92769-4"}]': finished 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:22.179505+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi029-92257-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:22.182090+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi029-92309-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:22.784620+0000 mon.a [INF] from='client.? 172.21.15.29:0/128313636' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi029-92698-12"}]: dispatch 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:22.919967+0000 mon.a [INF] from='client.? 172.21.15.29:0/911033316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi029-92291-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi029-92291-16"}]: dispatch 2023-08-22T23:25:32.365 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:23.613136+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi029-92365-10"}]: dispatch 2023-08-22T23:25:32.366 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:24.284021+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-08-22T23:25:32.366 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:24.337690+0000 mon.a [INF] from='client.? 172.21.15.29:0/128313636' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi029-92698-12"}]: dispatch 2023-08-22T23:25:35.856 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:23.607785+0000 mon.b [INF] from='client.? 172.21.15.29:0/1502112702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi029-92309-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi029-92309-22"}]: dispatch 2023-08-22T23:25:35.856 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:23.691544+0000 mon.b [INF] from='client.? 172.21.15.29:0/2003049875' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi029-92769-4", "tierpool": "test-rados-api-smithi029-92769-6"}]: dispatch 2023-08-22T23:25:35.856 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:23.863488+0000 mon.c [INF] from='client.4589 172.21.15.29:0/780599210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi029-92667-7"}]: dispatch 2023-08-22T23:25:35.856 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:24.755196+0000 mon.b [INF] from='client.? 172.21.15.29:0/4260781160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi029-92434-10"}]: dispatch 2023-08-22T23:25:35.856 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:25.686405+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi029-92309-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi029-92309-22"}]: dispatch 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:25.794497+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi029-92769-4", "tierpool": "test-rados-api-smithi029-92769-6"}]: dispatch 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:25.802711+0000 mon.a [INF] from='client.4589 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi029-92667-7"}]: dispatch 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:25.821906+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi029-92434-10"}]: dispatch 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:26.840002+0000 mon.a [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:26.893861+0000 mon.a [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:27.012189+0000 mon.a [INF] from='client.? 172.21.15.29:0/128313636' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi029-92698-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:27.442519+0000 mon.c [INF] from='client.? 172.21.15.29:0/2118062857' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-08-22T23:25:35.857 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:28.565519+0000 client.admin [INF] threexx 2023-08-22T23:25:35.858 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:28.600688+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-08-22T23:25:35.858 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:29.261096+0000 mon.a [INF] from='client.? 172.21.15.29:0/911033316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi029-92291-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi029-92291-16"}]': finished 2023-08-22T23:25:35.858 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:29.265932+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi029-92365-10"}]': finished 2023-08-22T23:25:35.858 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:29.273960+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi029-92769-4", "tierpool": "test-rados-api-smithi029-92769-6"}]': finished 2023-08-22T23:25:39.004 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:24.286 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.018 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:24.301 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:29.275943+0000 mon.a [INF] from='client.4589 ' entity='client.admin' cm api_stat: Running main() from gmock_main.cc 2023-08-22T23:25:39.194 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-08-22T23:25:39.194 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [----------] Global test environment set-up. 2023-08-22T23:25:39.194 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-08-22T23:25:39.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-08-22T23:25:39.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStat.Stat (24616 ms) 2023-08-22T23:25:39.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-08-22T23:25:39.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (41720 ms) 2023-08-22T23:25:39.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-08-22T23:25:39.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStat.StatNS (10139 ms) 2023-08-22T23:25:39.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-08-22T23:25:39.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (3 ms) 2023-08-22T23:25:39.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-08-22T23:25:39.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1344 ms) 2023-08-22T23:25:39.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [----------] 5 tests from LibRadosStat (77822 ms total) 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (14084 ms) 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (3228 ms) 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-08-22T23:25:39.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1188 ms) 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3479 ms) 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (21979 ms total) 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [----------] Global test environment tear-down 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (132153 ms total) 2023-08-22T23:25:39.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_stat: [ PASSED ] 9 tests. 2023-08-22T23:25:39.212 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-08-22T23:25:39.213 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-08-22T23:25:39.213 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-08-22T23:25:39.213 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-08-22T23:25:39.214 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: seed 92434 2023-08-22T23:25:39.214 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-08-22T23:25:39.214 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (40930 ms) 2023-08-22T23:25:39.214 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-08-22T23:25:39.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (10732 ms) 2023-08-22T23:25:39.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-08-22T23:25:39.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (14910 ms) 2023-08-22T23:25:39.215 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (2541 ms) 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4877 ms) 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (2573 ms) 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-08-22T23:25:39.216 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (386 ms) 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2702 ms) 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (79651 ms total) 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-08-22T23:25:39.217 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (6283 ms) 2023-08-22T23:25:39.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-08-22T23:25:39.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (237 ms) 2023-08-22T23:25:39.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-08-22T23:25:39.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2903 ms) 2023-08-22T23:25:39.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-08-22T23:25:39.218 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (2672 ms) 2023-08-22T23:25:39.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-08-22T23:25:39.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (806 ms) 2023-08-22T23:25:39.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-08-22T23:25:39.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (195 ms) 2023-08-22T23:25:39.219 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-08-22T23:25:39.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (339 ms) 2023-08-22T23:25:39.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-08-22T23:25:39.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (250 ms) 2023-08-22T23:25:39.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (13685 ms total) 2023-08-22T23:25:39.220 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: 2023-08-22T23:25:39.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-08-22T23:25:39.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (132262 ms total) 2023-08-22T23:25:39.221 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-08-22T23:25:39.328 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:04:24.611 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.488 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: Running main() from gmock_main.cc 2023-08-22T23:25:39.488 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-08-22T23:25:39.488 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [----------] Global test environment set-up. 2023-08-22T23:25:39.489 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-08-22T23:25:39.489 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-08-22T23:25:39.489 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (25687 ms) 2023-08-22T23:25:39.489 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-08-22T23:25:39.489 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.LockShared (2235 ms) 2023-08-22T23:25:39.490 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-08-22T23:25:39.490 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (5026 ms) 2023-08-22T23:25:39.490 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-08-22T23:25:39.490 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (8514 ms) 2023-08-22T23:25:39.491 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-08-22T23:25:39.491 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (9296 ms) 2023-08-22T23:25:39.491 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-08-22T23:25:39.491 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.Unlock (22000 ms) 2023-08-22T23:25:39.492 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-08-22T23:25:39.492 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (3152 ms) 2023-08-22T23:25:39.492 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-08-22T23:25:39.492 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (1548 ms) 2023-08-22T23:25:39.492 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [----------] 8 tests from LibRadosLock (77458 ms total) 2023-08-22T23:25:39.492 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: 2023-08-22T23:25:39.493 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-08-22T23:25:39.493 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-08-22T23:25:39.493 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (12053 ms) 2023-08-22T23:25:39.493 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-08-22T23:25:39.493 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (682 ms) 2023-08-22T23:25:39.493 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-08-22T23:25:39.494 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (2983 ms) 2023-08-22T23:25:39.494 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-08-22T23:25:39.494 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1158 ms) 2023-08-22T23:25:39.494 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-08-22T23:25:39.494 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (2131 ms) 2023-08-22T23:25:39.495 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-08-22T23:25:39.495 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (1280 ms) 2023-08-22T23:25:39.495 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-08-22T23:25:39.495 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (357 ms) 2023-08-22T23:25:39.495 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-08-22T23:25:39.495 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (293 ms) 2023-08-22T23:25:39.496 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (20937 ms total) 2023-08-22T23:25:39.496 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: 2023-08-22T23:25:39.496 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [----------] Global test environment tear-down 2023-08-22T23:25:39.496 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (132574 ms total) 2023-08-22T23:25:39.496 INFO:tasks.workunit.client.0.smithi029.stdout: api_lock: [ PASSED ] 16 tests. 2023-08-22T23:25:39.659 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:04:24.956 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.799 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:04:25.096 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.816 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:25.099 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.841 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:25.124 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:39.950 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:04:25.247 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:40.134 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:04:25.418 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:25:42.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: Running main() from gmock_main.cc 2023-08-22T23:25:42.326 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-08-22T23:25:42.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [----------] Global test environment set-up. 2023-08-22T23:25:42.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-08-22T23:25:42.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: seed 92309 2023-08-22T23:25:42.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-08-22T23:25:42.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-08-22T23:25:42.327 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (40519 ms) 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (4093 ms) 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (7156 ms) 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-08-22T23:25:42.328 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (3108 ms) 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (1029 ms) 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2725 ms) 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (7503 ms) 2023-08-22T23:25:42.329 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (1938 ms) 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (10640 ms) 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (1412 ms) 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4235 ms) 2023-08-22T23:25:42.330 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-08-22T23:25:42.331 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (2707 ms) 2023-08-22T23:25:42.331 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-08-22T23:25:42.331 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (317 ms) 2023-08-22T23:25:42.331 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-08-22T23:25:42.331 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (914 ms) 2023-08-22T23:25:42.332 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-08-22T23:25:42.332 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (1813 ms) 2023-08-22T23:25:42.332 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-08-22T23:25:42.332 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (217 ms) 2023-08-22T23:25:42.332 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-08-22T23:25:42.332 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (213 ms) 2023-08-22T23:25:42.333 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-08-22T23:25:42.333 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (63 ms) 2023-08-22T23:25:42.333 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-08-22T23:25:42.334 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (202 ms) 2023-08-22T23:25:42.334 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (90804 ms total) 2023-08-22T23:25:42.334 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: 2023-08-22T23:25:42.334 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-08-22T23:25:42.335 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-08-22T23:25:42.335 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (4594 ms) 2023-08-22T23:25:42.335 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-08-22T23:25:42.335 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (834 ms) 2023-08-22T23:25:42.336 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-08-22T23:25:42.336 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (283 ms) 2023-08-22T23:25:42.336 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-08-22T23:25:42.337 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (181 ms) 2023-08-22T23:25:42.337 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-08-22T23:25:42.374 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.Roun list_parallel: process_1_[92950]: starting. 2023-08-22T23:25:42.374 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_1_[92950]: creating pool ceph_test_rados_list_parallel.smithi029-92933 2023-08-22T23:25:42.374 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_1_[92950]: created object 0... 2023-08-22T23:25:42.374 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_1_[92950]: created object 25... 2023-08-22T23:25:42.374 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_1_[92950]: created object 49... 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_1_[92950]: finishing. 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_1_[92950]: shutting down. 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_2_[92951]: starting. 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_2_[92951]: listing objects. 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_2_[92951]: listed object 0... 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_2_[92951]: listed object 25... 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_2_[92951]: saw 50 objects 2023-08-22T23:25:42.375 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_2_[92951]: shutting down. 2023-08-22T23:25:42.376 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.376 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: starting. 2023-08-22T23:25:42.376 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: creating pool ceph_test_rados_list_parallel.smithi029-92933 2023-08-22T23:25:42.376 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: created object 0... 2023-08-22T23:25:42.376 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: created object 25... 2023-08-22T23:25:42.376 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: created object 49... 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: finishing. 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_3_[104316]: shutting down. 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_4_[104317]: starting. 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_4_[104317]: listing objects. 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_4_[104317]: listed object 0... 2023-08-22T23:25:42.377 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_4_[104317]: listed object 25... 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_4_[104317]: saw 39 objects 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_4_[104317]: shutting down. 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_5_[104318]: starting. 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_5_[104318]: removed 25 objects... 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_5_[104318]: removed half of the objects 2023-08-22T23:25:42.378 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_5_[104318]: removed 50 objects... 2023-08-22T23:25:42.379 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_5_[104318]: removed 50 objects 2023-08-22T23:25:42.380 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_5_[104318]: shutting down. 2023-08-22T23:25:42.380 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.380 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.381 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: starting. 2023-08-22T23:25:42.381 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: creating pool ceph_test_rados_list_parallel.smithi029-92933 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: created object 0... 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: created object 25... 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: created object 49... 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: finishing. 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_6_[118908]: shutting down. 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.382 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: starting. 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: listing objects. 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: listed object 0... 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: listed object 25... 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: listed object 50... 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: saw 57 objects 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_7_[118909]: shutting down. 2023-08-22T23:25:42.383 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_8_[118913]: starting. 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_8_[118913]: added 25 objects... 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_8_[118913]: added half of the objects 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_8_[118913]: added 50 objects... 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_8_[118913]: added 50 objects 2023-08-22T23:25:42.384 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_8_[118913]: shutting down. 2023-08-22T23:25:42.385 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:42.385 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:44.096 INFO:tasks.workunit.client.0.smithi029.stdout: id 433791696904 notifier_gid 4723 2023-08-22T23:25:44.096 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: timed out 2023-08-22T23:25:44.097 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: flushing 2023-08-22T23:25:44.097 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: flushed 2023-08-22T23:25:44.097 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3339 ms) 2023-08-22T23:25:44.098 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-08-22T23:25:44.098 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: trying... 2023-08-22T23:25:44.098 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578695536 notify_id 438086664195 notifier_gid 4723 2023-08-22T23:25:44.099 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: timed out 2023-08-22T23:25:44.099 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: flushing 2023-08-22T23:25:44.099 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: flushed 2023-08-22T23:25:44.099 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3189 ms) 2023-08-22T23:25:44.100 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-08-22T23:25:44.100 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: List watches 2023-08-22T23:25:44.100 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify2 2023-08-22T23:25:44.100 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578696576 notify_id 442381631498 notifier_gid 4723 2023-08-22T23:25:44.100 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify2 done 2023-08-22T23:25:44.101 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: watch_check 2023-08-22T23:25:44.101 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: unwatch2 2023-08-22T23:25:44.101 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: flushing 2023-08-22T23:25:44.101 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: done 2023-08-22T23:25:44.102 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3516 ms) 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: List watches 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify2 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: handle_notify cookie 94775578696576 notify_id 446676598795 notifier_gid 4723 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: notify2 done 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: watch_check 2023-08-22T23:25:44.104 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: unwatch2 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: flushing 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: done 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3443 ms) 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (57297 ms total) 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-08-22T23:25:44.105 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (137014 ms total) 2023-08-22T23:25:44.106 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-08-22T23:25:44.174 INFO:tasks.workunit.client.0.smithi029.stdout:d='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi029-92667-7"}]': finished 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:29.277744+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi029-92434-10"}]': finished 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: got: 2023-08-22T23:25:30.147163+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 20526.425ms) (OSD_SLOW_PING_TIME_BACK) 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (34178 ms) 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (137081 ms total) 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [----------] Global test environment tear-down 2023-08-22T23:25:44.175 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (137081 ms total) 2023-08-22T23:25:44.176 INFO:tasks.workunit.client.0.smithi029.stdout: api_cmd: [ PASSED ] 4 tests. 2023-08-22T23:25:48.482 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: starting. 2023-08-22T23:25:48.482 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: creating pool ceph_test_rados_open_pools_parallel.smithi029-93037 2023-08-22T23:25:48.482 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: created object 0... 2023-08-22T23:25:48.482 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: created object 25... 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: created object 49... 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: finishing. 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_1_[93064]: shutting down. 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_2_[93065]: starting. 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_2_[93065]: rados_pool_create. 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_2_[93065]: rados_ioctx_create. 2023-08-22T23:25:48.483 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_2_[93065]: shutting down. 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: ******************************* 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: starting. 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: creating pool ceph_test_rados_open_pools_parallel.smithi029-93037 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: created object 0... 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: created object 25... 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: created object 49... 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: finishing. 2023-08-22T23:25:48.484 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_3_[108539]: shutting down. 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: ******************************* 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_4_[108543]: starting. 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_4_[108543]: rados_pool_create. 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_4_[108543]: rados_ioctx_create. 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: process_4_[108543]: shutting down. 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: ******************************* 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: ******************************* 2023-08-22T23:25:48.485 INFO:tasks.workunit.client.0.smithi029.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-08-22T23:25:52.306 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: starting. 2023-08-22T23:25:52.307 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: creating pool ceph_test_rados_delete_pools_parallel.smithi029-93031 2023-08-22T23:25:52.307 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: created object 0... 2023-08-22T23:25:52.307 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: created object 25... 2023-08-22T23:25:52.307 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: created object 49... 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: finishing. 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_1_[93051]: shutting down. 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_2_[93053]: starting. 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_2_[93053]: deleting pool ceph_test_rados_delete_pools_parallel.smithi029-93031 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_2_[93053]: shutting down. 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: ******************************* 2023-08-22T23:25:52.308 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: starting. 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: creating pool ceph_test_rados_delete_pools_parallel.smithi029-93031 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: created object 0... 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: created object 25... 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: created object 49... 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: finishing. 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_3_[108399]: shutting down. 2023-08-22T23:25:52.309 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: ******************************* 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_5_[108401]: starting. 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_5_[108401]: listing objects. 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_5_[108401]: listed object 0... 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_5_[108401]: listed object 25... 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_5_[108401]: saw 50 objects 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_5_[108401]: shutting down. 2023-08-22T23:25:52.310 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: ******************************* 2023-08-22T23:25:52.311 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_4_[108400]: starting. 2023-08-22T23:25:52.311 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_4_[108400]: deleting pool ceph_test_rados_delete_pools_parallel.smithi029-93031 2023-08-22T23:25:52.311 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: process_4_[108400]: shutting down. 2023-08-22T23:25:52.311 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: ******************************* 2023-08-22T23:25:52.311 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: ******************************* 2023-08-22T23:25:52.311 INFO:tasks.workunit.client.0.smithi029.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-08-22T23:25:55.602 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: starting. 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: creating pool ceph_test_rados_list_parallel.smithi029-92933 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: created object 0... 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: created object 25... 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: created object 49... 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: finishing. 2023-08-22T23:25:55.603 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_9_[130813]: shutting down. 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: starting. 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: listing objects. 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: listed object 0... 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: listed object 25... 2023-08-22T23:25:55.604 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: listed object 50... 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: listed object 75... 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: saw 100 objects 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_10_[130814]: shutting down. 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_13_[130817]: starting. 2023-08-22T23:25:55.605 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_13_[130817]: removed 25 objects... 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_13_[130817]: removed half of the objects 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_13_[130817]: removed 50 objects... 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_13_[130817]: removed 50 objects 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_13_[130817]: shutting down. 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.606 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_12_[130816]: starting. 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_12_[130816]: added 25 objects... 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_12_[130816]: added half of the objects 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_12_[130816]: added 50 objects... 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_12_[130816]: added 50 objects 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_12_[130816]: shutting down. 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.607 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.608 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.608 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_11_[130815]: starting. 2023-08-22T23:25:55.608 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_11_[130815]: added 25 objects... 2023-08-22T23:25:55.608 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_11_[130815]: added half of the objects 2023-08-22T23:25:55.608 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_11_[130815]: added 50 objects... 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_11_[130815]: added 50 objects 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_11_[130815]: shutting down. 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: starting. 2023-08-22T23:25:55.609 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: creating pool ceph_test_rados_list_parallel.smithi029-92933 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: created object 0... 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: created object 25... 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: created object 49... 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: finishing. 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_14_[141190]: shutting down. 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.610 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.611 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.611 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:25:55.611 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: starting. 2023-08-22T23:25:55.611 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listing objects. 2023-08-22T23:26:05.409 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listed object 0... 2023-08-22T23:26:05.410 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listed object 25... 2023-08-22T23:26:05.410 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listed object 50... 2023-08-22T23:26:05.410 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listed object 75... 2023-08-22T23:26:05.410 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listed object 100... 2023-08-22T23:26:05.410 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: listed object 125... 2023-08-22T23:26:05.412 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: saw 150 objects 2023-08-22T23:26:05.412 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_15_[141191]: shutting down. 2023-08-22T23:26:05.413 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.413 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.413 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.414 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.414 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_16_[141192]: starting. 2023-08-22T23:26:05.414 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_16_[141192]: added 25 objects... 2023-08-22T23:26:05.414 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_16_[141192]: added half of the objects 2023-08-22T23:26:05.415 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_16_[141192]: added 50 objects... 2023-08-22T23:26:05.415 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_16_[141192]: added 50 objects 2023-08-22T23:26:05.415 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: process_16_[141192]: shutting down. 2023-08-22T23:26:05.415 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.416 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.416 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.416 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.417 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******************************* 2023-08-22T23:26:05.417 INFO:tasks.workunit.client.0.smithi029.stdout: list_parallel: ******* SUCCESS ********** 2023-08-22T23:26:05.417 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: Running main() from gmock_main.cc 2023-08-22T23:26:05.417 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-08-22T23:26:05.418 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [----------] Global test environment set-up. 2023-08-22T23:26:05.418 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-08-22T23:26:05.419 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-08-22T23:26:05.419 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (23172 ms) 2023-08-22T23:26:05.419 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-08-22T23:26:05.420 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-08-22T23:26:05.420 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-08-22T23:26:05.420 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: no timeout :/ 2023-08-22T23:26:05.421 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: no timeout :/ 2023-08-22T23:26:05.421 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: no timeout :/ 2023-08-22T23:26:05.422 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: no timeout :/ 2023-08-22T23:26:05.422 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (4815 ms) 2023-08-22T23:26:05.422 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-08-22T23:26:05.422 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (28577 ms) 2023-08-22T23:26:05.423 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-08-22T23:26:05.423 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.Checksum (12332 ms) 2023-08-22T23:26:05.423 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-08-22T23:26:05.423 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (2551 ms) 2023-08-22T23:26:05.424 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-08-22T23:26:05.424 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (12295 ms) 2023-08-22T23:26:05.424 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-08-22T23:26:05.425 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (3850 ms) 2023-08-22T23:26:05.425 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-08-22T23:26:05.425 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (291 ms) 2023-08-22T23:26:05.426 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-08-22T23:26:05.426 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.TruncTest (270 ms) 2023-08-22T23:26:05.426 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-08-22T23:26:05.426 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (977 ms) 2023-08-22T23:26:05.427 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-08-22T23:26:05.427 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (197 ms) 2023-08-22T23:26:05.427 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-08-22T23:26:05.427 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.RmXattr (493 ms) 2023-08-22T23:26:05.428 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-08-22T23:26:05.428 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIo.XattrIter (453 ms) 2023-08-22T23:26:05.428 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [----------] 14 tests from LibRadosIo (90273 ms total) 2023-08-22T23:26:05.428 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: 2023-08-22T23:26:05.429 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-08-22T23:26:05.429 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-08-22T23:26:05.429 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (5882 ms) 2023-08-22T23:26:05.429 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-08-22T23:26:05.430 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (388 ms) 2023-08-22T23:26:05.430 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-08-22T23:26:05.430 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (296 ms) 2023-08-22T23:26:05.431 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-08-22T23:26:05.431 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (264 ms) 2023-08-22T23:26:05.431 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-08-22T23:26:05.431 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (437 ms) 2023-08-22T23:26:05.432 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-08-22T23:26:05.432 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (322 ms) 2023-08-22T23:26:05.432 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-08-22T23:26:05.433 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (1239 ms) 2023-08-22T23:26:05.433 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-08-22T23:26:05.433 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (272 ms) 2023-08-22T23:26:05.433 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-08-22T23:26:05.434 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (764 ms) 2023-08-22T23:26:05.434 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-08-22T23:26:05.434 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (272 ms) 2023-08-22T23:26:05.434 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [----------] 10 tests from LibRadosIoEC (10136 ms total) 2023-08-22T23:26:05.435 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: 2023-08-22T23:26:05.435 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [----------] Global test environment tear-down 2023-08-22T23:26:05.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [==========] 24 tests from 2 test suites ran. (158540 ms total) 2023-08-22T23:26:05.436 INFO:tasks.workunit.client.0.smithi029.stdout: api_io: [ PASSED ] 24 tests. 2023-08-22T23:26:06.501 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-08-22T23:26:06.501 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] Global test environment set-up. 2023-08-22T23:26:06.501 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-08-22T23:26:06.501 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-08-22T23:26:06.502 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: 2023-08-22T23:23:27.398+0000 7fd930a32780 -1 failed for service _ceph-mon._tcp 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: 2023-08-22T23:23:27.398+0000 7fd930a32780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (464 ms) 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2005 ms) 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2469 ms total) 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: 2023-08-22T23:26:06.503 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae61695a0 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae57a6b70 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae56c0b20 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae5764260 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae58dc040 2023-08-22T23:26:06.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae58dca40 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae58dd4a0 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae58ddf20 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 0x560ae58de960 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: started 9 aios 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae61695a0 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae57a6b70 2023-08-22T23:26:06.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae56c0b20 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae5764260 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae58dc040 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae58dca40 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae58dd4a0 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae58ddf20 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: waiting 0x560ae58de960 2023-08-22T23:26:06.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: done. 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (16367 ms) 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (16367 ms total) 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-08-22T23:26:06.507 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.Exec (48785 ms) 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (8235 ms) 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (708 ms) 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-08-22T23:26:06.508 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6022 ms) 2023-08-22T23:26:06.509 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-08-22T23:26:06.509 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-08-22T23:26:06.509 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-08-22T23:26:06.509 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-08-22T23:26:06.509 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-08-22T23:26:06.510 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (71857 ms) 2023-08-22T23:26:06.510 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] 8 tests from LibRadosMisc (135607 ms total) 2023-08-22T23:26:06.510 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: 2023-08-22T23:26:06.510 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [----------] Global test environment tear-down 2023-08-22T23:26:06.511 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (159565 ms total) 2023-08-22T23:26:06.512 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc: [ PASSED ] 12 tests. 2023-08-22T23:26:06.711 INFO:tasks.workunit.client.0.smithi029.stdout:dTripPP2 (93 ms) 2023-08-22T23:26:06.712 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-08-22T23:26:06.712 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (370 ms) 2023-08-22T23:26:06.712 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-08-22T23:26:06.713 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (281 ms) 2023-08-22T23:26:06.713 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-08-22T23:26:06.713 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2028 ms) 2023-08-22T23:26:06.713 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-08-22T23:26:06.713 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (269 ms) 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (259 ms) 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (195 ms) 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (192 ms) 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-08-22T23:26:06.714 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (673 ms) 2023-08-22T23:26:06.715 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-08-22T23:26:06.715 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (244 ms) 2023-08-22T23:26:06.715 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-08-22T23:26:06.715 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (233 ms) 2023-08-22T23:26:06.715 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-08-22T23:26:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (107 ms) 2023-08-22T23:26:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-08-22T23:26:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (207 ms) 2023-08-22T23:26:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (11043 ms total) 2023-08-22T23:26:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: 2023-08-22T23:26:06.716 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [----------] Global test environment tear-down 2023-08-22T23:26:06.717 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (159836 ms total) 2023-08-22T23:26:06.717 INFO:tasks.workunit.client.0.smithi029.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-08-22T23:26:07.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-08-22T23:26:07.637 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-08-22T23:26:07.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-08-22T23:26:07.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-08-22T23:26:07.638 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-08-22T23:26:07.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify_test_cb 2023-08-22T23:26:07.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (69789 ms) 2023-08-22T23:26:07.639 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855004735824 err -107 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (8012 ms) 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855004735824 err -107 2023-08-22T23:26:07.640 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (6474 ms) 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 429496729600 cookie 93855007050768 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (946 ms) 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 429496729600 cookie 93855007050768 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (1509 ms) 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-08-22T23:26:07.641 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 429496729600 cookie 93855006446096 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (1586 ms) 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 433791696897 cookie 93855006446096 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 433791696897 cookie 93855007167632 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (646 ms) 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-08-22T23:26:07.642 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 433791696898 cookie 93855007167632 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 433791696903 cookie 93855007167632 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3402 ms) 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855007169808 err -107 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_cb from 4409 notify_id 446676598786 cookie 93855007169808 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (8028 ms) 2023-08-22T23:26:07.643 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855007169808 err -107 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1646 ms) 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (102038 ms total) 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-08-22T23:26:07.644 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-08-22T23:26:07.645 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: watch_notify_test_cb 2023-08-22T23:26:07.645 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (10791 ms) 2023-08-22T23:26:07.645 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (10791 ms total) 2023-08-22T23:26:07.645 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: 2023-08-22T23:26:07.645 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-08-22T23:26:07.645 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (160591 ms total) 2023-08-22T23:26:07.646 INFO:tasks.workunit.client.0.smithi029.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-08-22T23:26:32.768 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:18.051 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:32.827 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:18.124 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:32.828 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:18.126 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:32.843 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:18.126 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:32.844 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:18.127 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:32.904 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:18.187 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:32.980 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:18.277 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:33.018 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:18.301 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:33.081 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:18.362 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:38.696 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: Running main() from gmock_main.cc 2023-08-22T23:26:38.696 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-08-22T23:26:38.697 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] Global test environment set-up. 2023-08-22T23:26:38.697 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-08-22T23:26:38.697 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-08-22T23:26:38.698 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (25765 ms) 2023-08-22T23:26:38.698 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-08-22T23:26:38.698 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3870 ms) 2023-08-22T23:26:38.698 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-08-22T23:26:38.699 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (16266 ms) 2023-08-22T23:26:38.699 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-08-22T23:26:38.699 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (6837 ms) 2023-08-22T23:26:38.699 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (52738 ms total) 2023-08-22T23:26:38.700 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: 2023-08-22T23:26:38.701 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-08-22T23:26:38.702 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-08-22T23:26:38.702 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (11792 ms) 2023-08-22T23:26:38.702 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-08-22T23:26:38.702 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (11703 ms) 2023-08-22T23:26:38.703 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-08-22T23:26:38.703 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (31740 ms) 2023-08-22T23:26:38.703 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (55235 ms total) 2023-08-22T23:26:38.703 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: 2023-08-22T23:26:38.704 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-08-22T23:26:38.704 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-08-22T23:26:38.704 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (7537 ms) 2023-08-22T23:26:38.704 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-08-22T23:26:38.705 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2050 ms) 2023-08-22T23:26:38.705 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-08-22T23:26:38.706 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (3806 ms) 2023-08-22T23:26:38.708 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-08-22T23:26:38.708 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2249 ms) 2023-08-22T23:26:38.708 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (15642 ms total) 2023-08-22T23:26:38.709 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: 2023-08-22T23:26:38.709 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-08-22T23:26:38.709 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-08-22T23:26:38.710 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (5693 ms) 2023-08-22T23:26:38.710 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-08-22T23:26:38.711 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (7610 ms) 2023-08-22T23:26:38.711 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (13304 ms total) 2023-08-22T23:26:38.711 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: 2023-08-22T23:26:38.711 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [----------] Global test environment tear-down 2023-08-22T23:26:38.712 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (191688 ms total) 2023-08-22T23:26:38.713 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-08-22T23:26:52.729 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:38.011 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.731 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:38.014 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.785 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:38.069 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.788 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:38.072 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.790 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:38.087 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.791 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:38.074 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.794 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:38.091 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.852 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:38.149 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:52.859 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:38.141 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:26:57.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-08-22T23:26:57.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-08-22T23:26:57.055 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: seed 92591 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (17049 ms) 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (9812 ms) 2023-08-22T23:26:57.056 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (11651 ms) 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (6679 ms) 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (20441 ms) 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (65632 ms total) 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: 2023-08-22T23:26:57.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (12413 ms) 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (35561 ms) 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (40592 ms) 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-08-22T23:26:57.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3674 ms) 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2904 ms) 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-5573-gf0ed7046/rpm/el9/BUILD/ceph-18.0.0-5573-gf0ed7046/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-08-22T23:26:57.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi029-92591-7 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: waiting for snaps to purge 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17777 ms) 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (112921 ms total) 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-08-22T23:26:57.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (4150 ms) 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2122 ms) 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3231 ms) 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2725 ms) 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (12228 ms total) 2023-08-22T23:26:57.061 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: 2023-08-22T23:26:57.062 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-08-22T23:27:13.066 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:58.349 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.068 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:58.352 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.141 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:58.438 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.147 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:58.444 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.151 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:05:58.435 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.165 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:58.448 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.171 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:58.454 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.221 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:58.518 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:13.264 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:05:58.548 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:18.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-08-22T23:27:18.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (6246 ms) 2023-08-22T23:27:18.504 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (7857 ms) 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4247 ms) 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (18350 ms total) 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-08-22T23:27:18.505 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (231499 ms total) 2023-08-22T23:27:18.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-08-22T23:27:18.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-08-22T23:27:18.506 INFO:tasks.workunit.client.0.smithi029.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-08-22T23:27:22.966 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: Running main() from gmock_main.cc 2023-08-22T23:27:22.966 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-08-22T23:27:22.966 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [----------] Global test environment set-up. 2023-08-22T23:27:22.966 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-08-22T23:27:22.966 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5799 ms) 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (25576 ms) 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (34586 ms) 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (8326 ms) 2023-08-22T23:27:22.967 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (25651 ms) 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (27625 ms) 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (28819 ms) 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (6411 ms) 2023-08-22T23:27:22.968 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4857 ms) 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (6038 ms) 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4166 ms) 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3781 ms) 2023-08-22T23:27:22.969 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3129 ms) 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4789 ms) 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.Flush (4318 ms) 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-08-22T23:27:22.970 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4244 ms) 2023-08-22T23:27:22.971 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-08-22T23:27:22.971 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3833 ms) 2023-08-22T23:27:22.971 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-08-22T23:27:22.971 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4645 ms) 2023-08-22T23:27:22.971 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-08-22T23:27:22.971 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4463 ms) 2023-08-22T23:27:22.972 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-08-22T23:27:22.972 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3408 ms) 2023-08-22T23:27:22.972 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-08-22T23:27:22.972 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3906 ms) 2023-08-22T23:27:22.972 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-08-22T23:27:22.972 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3729 ms) 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4034 ms) 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3403 ms) 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3201 ms) 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-08-22T23:27:22.973 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3357 ms) 2023-08-22T23:27:22.974 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [----------] 26 tests from LibRadosAio (236094 ms total) 2023-08-22T23:27:22.974 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: 2023-08-22T23:27:22.974 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-08-22T23:27:28.527 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleW api_aio_pp: Running main() from gmock_main.cc 2023-08-22T23:27:28.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-08-22T23:27:28.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-08-22T23:27:28.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-08-22T23:27:28.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-08-22T23:27:28.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (5784 ms) 2023-08-22T23:27:28.528 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (38761 ms) 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (39178 ms) 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (12136 ms) 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (28045 ms) 2023-08-22T23:27:28.529 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (30117 ms) 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (8775 ms) 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3825 ms) 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3412 ms) 2023-08-22T23:27:28.530 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-08-22T23:27:28.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3685 ms) 2023-08-22T23:27:28.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-08-22T23:27:28.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4131 ms) 2023-08-22T23:27:28.531 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3778 ms) 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3172 ms) 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4779 ms) 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-08-22T23:27:28.532 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (6384 ms) 2023-08-22T23:27:28.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-08-22T23:27:28.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3133 ms) 2023-08-22T23:27:28.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-08-22T23:27:28.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2842 ms) 2023-08-22T23:27:28.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-08-22T23:27:28.533 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4679 ms) 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (4444 ms) 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3446 ms) 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (4732 ms) 2023-08-22T23:27:28.534 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-08-22T23:27:28.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (5026 ms) 2023-08-22T23:27:28.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-08-22T23:27:28.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3088 ms) 2023-08-22T23:27:28.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-08-22T23:27:28.535 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4273 ms) 2023-08-22T23:27:28.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-08-22T23:27:28.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3184 ms) 2023-08-22T23:27:28.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-08-22T23:27:28.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3311 ms) 2023-08-22T23:27:28.536 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-08-22T23:27:38.062 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:23.345 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.065 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:23.348 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.131 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:23.414 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.133 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:23.430 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.134 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:23.417 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.135 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:23.432 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.135 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:23.417 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.205 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:23.489 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:38.253 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:23.550 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:50.339 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadoist: : entry=8 expected=8 2023-08-22T23:27:50.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:b29083e3:::5:head 2023-08-22T23:27:50.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:b29083e3:::5:head expected=11:b29083e3:::5:head 2023-08-22T23:27:50.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:b29083e3:::5:head -> 5 2023-08-22T23:27:50.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=5 expected=5 2023-08-22T23:27:50.340 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:89d3ae78:::11:head 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:89d3ae78:::11:head expected=11:89d3ae78:::11:head 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=11 expected=11 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:863748b0:::15:head 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:863748b0:::15:head expected=11:863748b0:::15:head 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:863748b0:::15:head -> 15 2023-08-22T23:27:50.341 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=15 expected=15 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:02547ec2:::1:head 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:02547ec2:::1:head expected=11:02547ec2:::1:head 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:02547ec2:::1:head -> 1 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=1 expected=1 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:d83876eb:::4:head 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:d83876eb:::4:head expected=11:d83876eb:::4:head 2023-08-22T23:27:50.342 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:d83876eb:::4:head -> 4 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=4 expected=4 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:cfc208b3:::3:head 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:cfc208b3:::3:head expected=11:cfc208b3:::3:head 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=3 expected=3 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2023-08-22T23:27:50.343 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:c4fdafeb:::6:head expected=11:c4fdafeb:::6:head 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=6 expected=6 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : seek to 11:566253c9:::13:head 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : cursor()=11:566253c9:::13:head expected=11:566253c9:::13:head 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: > 11:566253c9:::13:head -> 13 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: : entry=13 expected=13 2023-08-22T23:27:50.344 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (17055 ms) 2023-08-22T23:27:50.345 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-08-22T23:27:50.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (88253 ms) 2023-08-22T23:27:50.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-08-22T23:27:50.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: split 0/5 -> MIN 11:33333333::::head 2023-08-22T23:27:50.346 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: split 1/5 -> 11:33333333::::head 11:66666666::::head 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: split 2/5 -> 11:66666666::::head 11:99999999::::head 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: split 3/5 -> 11:99999999::::head 11:cccccccc::::head 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: split 4/5 -> 11:cccccccc::::head MAX 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (37175 ms) 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] 7 tests from LibRadosList (248346 ms total) 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-08-22T23:27:50.347 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (4210 ms) 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset foo1,foo2,foo3 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo1 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo2 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo3 2023-08-22T23:27:50.348 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset foo1,foo4,foo5 2023-08-22T23:27:50.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo4 2023-08-22T23:27:50.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo5 2023-08-22T23:27:50.349 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo1 2023-08-22T23:27:50.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset foo6,foo7 2023-08-22T23:27:50.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo7 2023-08-22T23:27:50.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: foo6 2023-08-22T23:27:50.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-08-22T23:27:50.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns1:foo4 2023-08-22T23:27:50.350 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns1:foo5 2023-08-22T23:27:50.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns2:foo7 2023-08-22T23:27:50.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns2:foo6 2023-08-22T23:27:50.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: ns1:foo1 2023-08-22T23:27:50.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: :foo1 2023-08-22T23:27:50.351 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: :foo2 2023-08-22T23:27:50.352 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: :foo3 2023-08-22T23:27:50.352 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3898 ms) 2023-08-22T23:27:57.194 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRa api_c_write_operations: Running main() from gmock_main.cc 2023-08-22T23:27:57.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-08-22T23:27:57.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-08-22T23:27:57.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-08-22T23:27:57.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-08-22T23:27:57.195 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-08-22T23:27:57.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-08-22T23:27:57.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: 2023-08-22T23:27:57.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-08-22T23:27:57.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-08-22T23:27:57.196 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (21269 ms) 2023-08-22T23:27:57.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-08-22T23:27:57.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (9805 ms) 2023-08-22T23:27:57.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-08-22T23:27:57.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (13213 ms) 2023-08-22T23:27:57.197 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-08-22T23:27:57.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (15334 ms) 2023-08-22T23:27:57.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-08-22T23:27:57.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (27654 ms) 2023-08-22T23:27:57.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-08-22T23:27:57.198 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (33482 ms) 2023-08-22T23:27:57.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-08-22T23:27:57.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (149306 ms) 2023-08-22T23:27:57.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (270063 ms total) 2023-08-22T23:27:57.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: 2023-08-22T23:27:57.199 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-08-22T23:27:57.200 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (270063 ms total) 2023-08-22T23:27:57.200 INFO:tasks.workunit.client.0.smithi029.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-08-22T23:27:58.070 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:43.353 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.129 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:43.427 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.137 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:43.434 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.144 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:43.428 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.184 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:43.481 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.189 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:43.472 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.192 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:43.475 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.230 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:06:43.513 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:27:58.258 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:06:43.541 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:01.874 INFO:tasks.workunit.client.0.smithi029.stdout:dosListEC.ListObjectsStart 2023-08-22T23:28:01.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 1 0 2023-08-22T23:28:01.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 10 0 2023-08-22T23:28:01.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 13 0 2023-08-22T23:28:01.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 7 0 2023-08-22T23:28:01.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 14 0 2023-08-22T23:28:01.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 0 0 2023-08-22T23:28:01.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 15 0 2023-08-22T23:28:01.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 11 0 2023-08-22T23:28:01.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 5 0 2023-08-22T23:28:01.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 8 0 2023-08-22T23:28:01.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 6 0 2023-08-22T23:28:01.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 3 0 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 4 0 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 12 0 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 9 0 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 2 0 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (4216 ms) 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] 3 tests from LibRadosListEC (12324 ms total) 2023-08-22T23:28:01.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (4550 ms) 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] 1 test from LibRadosListNP (4550 ms total) 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [----------] Global test environment tear-down 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [==========] 11 tests from 3 test suites ran. (274975 ms total) 2023-08-22T23:28:01.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_list: [ PASSED ] 11 tests. 2023-08-22T23:28:11.838 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:11.839 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: seed 92451 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (200 ms) 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (57225 ms) 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-08-22T23:28:11.840 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (10381 ms) 2023-08-22T23:28:11.841 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-08-22T23:28:11.841 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (8453 ms) 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (1922 ms) 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (826 ms) 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2127 ms) 2023-08-22T23:28:11.842 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5380 ms) 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1126 ms) 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (635 ms) 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (786 ms) 2023-08-22T23:28:11.843 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-08-22T23:28:11.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (739 ms) 2023-08-22T23:28:11.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-08-22T23:28:11.844 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (301 ms) 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: osd_max_attr_size = 0 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (87335 ms) 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (15841 ms) 2023-08-22T23:28:11.845 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-08-22T23:28:11.846 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-08-22T23:28:11.846 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: done waiting, doing copies 2023-08-22T23:28:11.846 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: waiting for final deep scrubs... 2023-08-22T23:28:11.846 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: done waiting 2023-08-22T23:28:11.847 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (82226 ms) 2023-08-22T23:28:11.847 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-08-22T23:28:11.848 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (475 ms) 2023-08-22T23:28:11.848 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-08-22T23:28:11.848 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (76 ms) 2023-08-22T23:28:11.848 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-08-22T23:28:11.849 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4566 ms) 2023-08-22T23:28:11.849 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-08-22T23:28:11.849 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-08-22T23:28:11.849 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-08-22T23:28:11.850 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-08-22T23:28:11.850 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-08-22T23:28:11.850 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-08-22T23:28:18.330 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:03.613 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.336 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:03.619 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.384 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:03.681 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.391 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:03.673 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.399 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:03.683 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.408 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:03.705 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.420 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:03.703 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.456 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:03.739 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:18.468 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:03.765 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.480 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:23.763 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.483 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:23.766 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.530 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:23.813 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.533 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:23.817 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.538 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:23.821 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.542 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:23.839 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.543 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:23.841 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.584 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:23.867 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:38.591 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:23.888 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.505 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:26.789 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.511 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:26.795 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.531 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:26.815 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.568 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:26.865 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.569 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:26.866 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.586 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:26.870 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.587 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:26.871 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.588 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:26.885 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:41.608 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:26.891 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:45.689 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (280620 ms total) 2023-08-22T23:28:45.722 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:45.722 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-08-22T23:28:45.723 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-08-22T23:28:45.723 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (4664 ms) 2023-08-22T23:28:45.723 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (4664 ms total) 2023-08-22T23:28:45.723 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:45.723 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-08-22T23:28:45.723 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-08-22T23:28:45.724 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (504 ms) 2023-08-22T23:28:45.724 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-08-22T23:28:45.724 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (37 ms) 2023-08-22T23:28:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (541 ms total) 2023-08-22T23:28:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-08-22T23:28:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-08-22T23:28:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (451 ms) 2023-08-22T23:28:45.725 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (445 ms) 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (896 ms total) 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (424 ms) 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-08-22T23:28:45.726 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (50 ms) 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (474 ms total) 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1840 ms) 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1840 ms total) 2023-08-22T23:28:45.727 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: 2023-08-22T23:28:45.728 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-08-22T23:28:45.728 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (318724 ms total) 2023-08-22T23:28:45.728 INFO:tasks.workunit.client.0.smithi029.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-08-22T23:28:58.602 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:43.886 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.606 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:43.889 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.660 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:43.957 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.661 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:43.958 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.677 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:43.960 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.678 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:43.961 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.689 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:07:43.972 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.698 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:43.995 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:28:58.714 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:07:43.997 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.772 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:04.056 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.823 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:04.106 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.824 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:04.107 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.828 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:04.112 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.830 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:04.127 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.833 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:04.131 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.874 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:04.158 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.879 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:04.176 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:18.961 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:04.244 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:23.921 INFO:tasks.workunit.client.0.smithi029.stdout:rite 2023-08-22T23:29:23.921 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7893 ms) 2023-08-22T23:29:23.921 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-08-22T23:29:23.921 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6761 ms) 2023-08-22T23:29:23.922 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-08-22T23:29:23.922 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7890 ms) 2023-08-22T23:29:23.922 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-08-22T23:29:23.922 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7229 ms) 2023-08-22T23:29:23.922 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-08-22T23:29:23.923 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (9143 ms) 2023-08-22T23:29:23.923 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-08-22T23:29:23.923 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6743 ms) 2023-08-22T23:29:23.923 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-08-22T23:29:23.924 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8027 ms) 2023-08-22T23:29:23.924 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-08-22T23:29:23.924 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6848 ms) 2023-08-22T23:29:23.924 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-08-22T23:29:23.924 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7771 ms) 2023-08-22T23:29:23.924 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-08-22T23:29:23.925 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6554 ms) 2023-08-22T23:29:23.925 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-08-22T23:29:23.925 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7863 ms) 2023-08-22T23:29:23.925 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-08-22T23:29:23.925 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7290 ms) 2023-08-22T23:29:23.925 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-08-22T23:29:23.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7420 ms) 2023-08-22T23:29:23.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-08-22T23:29:23.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7927 ms) 2023-08-22T23:29:23.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-08-22T23:29:23.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7699 ms) 2023-08-22T23:29:23.926 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-08-22T23:29:23.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7895 ms) 2023-08-22T23:29:23.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (120954 ms total) 2023-08-22T23:29:23.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: 2023-08-22T23:29:23.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [----------] Global test environment tear-down 2023-08-22T23:29:23.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (357048 ms total) 2023-08-22T23:29:23.927 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio: [ PASSED ] 42 tests. 2023-08-22T23:29:38.936 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:24.219 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:38.938 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:24.221 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:38.978 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:24.261 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:38.983 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:24.267 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:38.984 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:24.281 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:38.986 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:24.283 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:38.988 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:24.272 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:39.026 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:24.323 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:39.034 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:24.317 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.030 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:44.313 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.032 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:44.316 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.071 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:08:44.355 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.077 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:44.360 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.078 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:44.376 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.081 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:44.365 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.082 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:44.379 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.124 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:08:44.407 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:29:59.124 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:44.421 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:04.146 INFO:tasks.workunit.client.0.smithi029.stdout:sAio.RoundTripAppendPP (3539 ms) 2023-08-22T23:30:04.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-08-22T23:30:04.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3355 ms) 2023-08-22T23:30:04.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-08-22T23:30:04.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (4646 ms) 2023-08-22T23:30:04.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-08-22T23:30:04.147 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (4479 ms) 2023-08-22T23:30:04.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-08-22T23:30:04.148 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: setting pool EIO 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: max_success 0, min_failed 1 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5731 ms) 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3675 ms) 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (263550 ms total) 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: 2023-08-22T23:30:04.149 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3399 ms) 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9465 ms) 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16576 ms) 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-08-22T23:30:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3076 ms) 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (32517 ms total) 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2561 ms) 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2561 ms total) 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: 2023-08-22T23:30:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15717 ms) 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7548 ms) 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7068 ms) 2023-08-22T23:30:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7625 ms) 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3809 ms) 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6804 ms) 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7620 ms) 2023-08-22T23:30:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6452 ms) 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6822 ms) 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7819 ms) 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6697 ms) 2023-08-22T23:30:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-08-22T23:30:04.155 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7869 ms) 2023-08-22T23:30:04.155 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-08-22T23:30:04.155 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6804 ms) 2023-08-22T23:30:19.243 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:04.526 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.313 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:04.596 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.340 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:04.624 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.341 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:04.639 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.342 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:04.625 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.372 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:04.669 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.406 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:04.689 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.406 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:04.690 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:19.439 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:04.736 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.584 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:14.867 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.588 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:14.872 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.622 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:14.906 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.654 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:14.937 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.657 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:14.955 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.660 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:14.957 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.661 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:14.945 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.695 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:14.978 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:29.696 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:14.993 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.410 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:24.693 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.425 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:24.707 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.465 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:24.747 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.484 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:24.767 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.496 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:24.789 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.500 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:24.797 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.500 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:24.783 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.527 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:24.824 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:39.552 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:24.835 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:50.519 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3488 ms) 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7644 ms) 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7108 ms) 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-08-22T23:30:50.520 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7090 ms) 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7393 ms) 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6603 ms) 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7044 ms) 2023-08-22T23:30:50.521 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (145025 ms total) 2023-08-22T23:30:50.522 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: 2023-08-22T23:30:50.522 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-08-22T23:30:50.522 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (443653 ms total) 2023-08-22T23:30:50.522 INFO:tasks.workunit.client.0.smithi029.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-08-22T23:30:59.272 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:44.555 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.276 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:44.560 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.307 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:09:44.590 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.323 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:44.620 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.327 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:44.611 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.330 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:44.613 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.343 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:44.641 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.361 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:09:44.645 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:30:59.376 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:44.673 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.422 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:04.706 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.485 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:04.768 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.486 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:04.769 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.488 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:04.771 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.490 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:04.787 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.504 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:04.801 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.517 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:04.800 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.530 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:04.828 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:19.549 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:04.829 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.554 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:24.837 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.560 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:24.844 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.604 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:24.887 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.620 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:24.917 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.625 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:24.908 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.625 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:24.922 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.633 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:24.917 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.678 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:24.962 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:39.679 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:24.976 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.599 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:44.881 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.605 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:44.886 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.611 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:44.893 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.612 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:44.909 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.613 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:44.895 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.658 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:44.955 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.691 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:10:44.974 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.698 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:10:44.981 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:31:59.725 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:45.022 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:12.870 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-08-22T23:32:12.870 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-08-22T23:32:12.871 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-08-22T23:32:12.871 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: seed 92481 2023-08-22T23:32:12.871 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-08-22T23:32:12.871 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (26462 ms) 2023-08-22T23:32:12.871 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-08-22T23:32:12.872 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (85302 ms) 2023-08-22T23:32:12.872 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-08-22T23:32:12.872 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (26 ms) 2023-08-22T23:32:12.872 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (111790 ms total) 2023-08-22T23:32:12.872 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (27727 ms) 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (21845 ms) 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (27437 ms) 2023-08-22T23:32:12.873 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: my_snaps [4] 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: my_snaps [5,4] 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: my_snaps [6,5,4] 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: promoting some heads 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: promoting from clones for snap 7 2023-08-22T23:32:12.874 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: promoting from clones for snap 6 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: promoting from clones for snap 5 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: promoting from clones for snap 4 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: waiting for scrubs... 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: done waiting 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (138937 ms) 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9954 ms) 2023-08-22T23:32:12.875 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-08-22T23:32:12.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10040 ms) 2023-08-22T23:32:12.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-08-22T23:32:12.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8883 ms) 2023-08-22T23:32:12.876 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10019 ms) 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12664 ms) 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10055 ms) 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-08-22T23:32:12.877 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11198 ms) 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40907 ms) 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10115 ms) 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10272 ms) 2023-08-22T23:32:12.878 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-08-22T23:32:12.879 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13980 ms) 2023-08-22T23:32:12.879 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-08-22T23:32:12.879 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8728 ms) 2023-08-22T23:32:12.879 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-08-22T23:32:12.879 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (10994 ms) 2023-08-22T23:32:19.692 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:04.975 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.694 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:04.978 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.727 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:05.011 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.744 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:05.027 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.744 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:05.042 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.745 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:05.029 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.749 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:05.046 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.781 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:05.078 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:19.788 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:05.071 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.676 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:18.959 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.679 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:18.962 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.710 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:18.994 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.730 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:19.027 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.737 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:19.030 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.738 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:19.019 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.746 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:19.021 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.780 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:19.062 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:33.785 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:19.081 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:35.267 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-08-22T23:32:35.268 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: hmm, no HitSet yet 2023-08-22T23:32:35.268 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-08-22T23:32:35.268 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10052 ms) 2023-08-22T23:32:35.268 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-08-22T23:32:35.268 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg_num = 32 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 0 ls 1692747150,0 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 1 ls 1692747150,0 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 2 ls 1692747150,0 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 3 ls 1692747150,0 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 4 ls 1692747150,0 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 5 ls 1692747150,0 2023-08-22T23:32:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 6 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 7 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 8 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 9 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 10 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 11 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 12 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 13 ls 1692747150,0 2023-08-22T23:32:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 14 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 15 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 16 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 17 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 18 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 19 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 20 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 21 ls 1692747150,0 2023-08-22T23:32:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 22 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 23 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 24 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 25 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 26 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 27 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 28 ls 1692747150,0 2023-08-22T23:32:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 29 ls 1692747150,0 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 30 ls 1692747150,0 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg 31 ls 1692747150,0 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: pg_num = 32 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-08-22T23:32:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-08-22T23:32:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-08-22T23:32:35.275 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-08-22T23:32:35.276 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-08-22T23:32:35.277 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-08-22T23:32:35.277 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-08-22T23:32:35.277 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-08-22T23:32:35.277 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-08-22T23:32:35.278 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-08-22T23:32:35.279 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-08-22T23:32:35.280 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-08-22T23:32:35.281 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-08-22T23:32:35.282 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-08-22T23:32:35.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-08-22T23:32:35.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-08-22T23:32:35.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-08-22T23:32:35.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-08-22T23:32:35.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-08-22T23:32:35.283 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-08-22T23:32:35.284 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-08-22T23:32:35.284 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-08-22T23:32:35.285 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-08-22T23:32:35.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-08-22T23:32:35.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-08-22T23:32:35.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-08-22T23:32:35.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-08-22T23:32:35.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-08-22T23:32:35.286 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-08-22T23:32:35.287 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-08-22T23:32:35.288 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-08-22T23:32:35.289 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-08-22T23:32:35.290 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-08-22T23:32:35.291 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-08-22T23:32:35.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-08-22T23:32:35.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-08-22T23:32:35.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-08-22T23:32:35.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-08-22T23:32:35.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-08-22T23:32:35.292 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-08-22T23:32:35.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-08-22T23:32:35.295 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-08-22T23:32:35.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-08-22T23:32:35.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-08-22T23:32:35.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-08-22T23:32:35.296 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-08-22T23:32:35.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-08-22T23:32:35.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-08-22T23:32:35.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-08-22T23:32:35.297 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-08-22T23:32:35.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-08-22T23:32:35.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-08-22T23:32:35.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-08-22T23:32:35.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-08-22T23:32:35.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-08-22T23:32:35.298 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-08-22T23:32:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-08-22T23:32:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-08-22T23:32:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-08-22T23:32:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-08-22T23:32:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-08-22T23:32:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-08-22T23:32:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-08-22T23:32:39.805 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:25.089 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.808 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:25.092 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.823 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:25.106 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.855 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:25.152 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.856 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:25.138 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.857 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:25.141 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.864 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:25.161 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.872 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:25.155 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:39.877 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:25.174 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.761 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:45.044 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.763 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:45.047 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.791 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:11:45.074 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.812 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:45.109 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.813 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:45.097 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.815 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:45.098 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.816 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:45.114 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.842 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:11:45.125 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:32:59.857 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:45.142 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.003 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:05.286 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.007 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:05.290 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.045 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:05.328 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.068 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:05.352 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.071 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:05.354 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.076 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:05.372 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.100 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:05.397 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.104 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:05.388 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:20.132 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:05.429 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.897 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:10.180 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.900 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:10.184 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.915 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:10.199 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.948 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:10.231 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.950 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:10.234 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.953 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:10.250 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.957 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:10.254 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.964 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:10.248 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:24.970 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:10.267 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:39.956 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:25.239 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:39.962 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:25.246 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:39.991 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:25.275 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.013 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:25.297 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.014 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:25.311 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.017 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:25.314 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.017 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:25.301 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.045 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:25.342 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.046 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:25.329 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.709 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:26.007 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.710 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:26.008 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.718 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:26.015 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.729 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:26.013 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.730 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:26.014 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.735 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:26.018 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.736 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:26.020 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.741 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:26.024 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:40.748 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:26.031 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:50.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-08-22T23:33:50.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-08-22T23:33:50.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-08-22T23:33:50.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-08-22T23:33:50.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-08-22T23:33:50.057 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-08-22T23:33:50.058 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-08-22T23:33:50.059 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-08-22T23:33:50.060 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-08-22T23:33:50.071 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-08-22T23:33:50.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-08-22T23:33:50.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-08-22T23:33:50.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-08-22T23:33:50.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-08-22T23:33:50.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-08-22T23:33:50.072 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22804 ms) 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,0 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: first is 1692747174 2023-08-22T23:33:50.073 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,1692747180,1692747181,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,1692747180,1692747181,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,1692747180,1692747181,0 2023-08-22T23:33:50.074 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,1692747180,1692747181,1692747183,1692747184,0 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,1692747180,1692747181,1692747183,1692747184,0 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747174,1692747176,1692747178,1692747180,1692747181,1692747183,1692747184,0 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747178,1692747180,1692747181,1692747183,1692747184,1692747186,1692747187,0 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: first now 1692747178, trimmed 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (24089 ms) 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-08-22T23:33:50.075 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: foo0 2023-08-22T23:33:50.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-08-22T23:33:50.076 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (18120 ms) 2023-08-22T23:33:50.078 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-08-22T23:33:55.066 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:40.349 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.067 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:40.351 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.103 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:40.387 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.117 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:40.414 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.118 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:40.416 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.129 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:40.413 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.131 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:40.414 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.140 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:40.438 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:33:55.154 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:40.438 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.267 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:45.550 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.279 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:45.554 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.304 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:45.578 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.316 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:45.600 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.319 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:45.603 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.325 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:45.622 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.331 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:45.628 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.339 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:45.622 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:00.347 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:45.643 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.155 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:55.436 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.156 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:55.439 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.171 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:12:55.454 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.203 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:55.485 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.203 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:55.486 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.209 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:55.507 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.211 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:55.508 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.220 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:12:55.503 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:10.232 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:55.523 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.186 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:05.469 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.195 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:05.473 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.214 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:05.497 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.254 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:05.534 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.254 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:05.536 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.265 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:05.521 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.265 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:05.524 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.265 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:05.547 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:20.266 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:05.557 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.205 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:15.489 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.273 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:15.570 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.278 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:15.572 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.280 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:15.564 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.284 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:15.567 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.285 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:15.569 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.304 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:15.601 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.308 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:15.592 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:30.330 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:15.614 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.295 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:25.592 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.296 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:25.593 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.306 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:25.589 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.316 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:25.596 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.323 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:25.620 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.355 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:25.639 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.383 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:25.665 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.386 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:25.669 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:40.414 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:25.697 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.298 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:30.581 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.302 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:30.585 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.318 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:30.600 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.347 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:30.644 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.348 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:30.631 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.349 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:30.633 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.350 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:30.648 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.365 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:30.648 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:34:45.365 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:30.661 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.356 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:45.640 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.359 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:45.642 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.385 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:45.666 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.413 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:45.694 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.414 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:45.695 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.415 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:45.712 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.421 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:45.714 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.436 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:45.720 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:00.443 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:45.739 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.110 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:46.407 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.112 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:46.408 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.120 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:46.417 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.156 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:46.439 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.157 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:46.441 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.169 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:13:46.452 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.170 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:46.453 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.174 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:46.454 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:01.195 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:13:46.463 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.422 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:05.706 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.425 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:05.709 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.448 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:05.731 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.474 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:05.772 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.475 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:05.758 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.476 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:05.760 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.481 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:05.778 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.498 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:05.782 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:20.502 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:05.799 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.330 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:06.628 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.332 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:06.629 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.344 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:06.638 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.399 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:06.682 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.400 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:06.684 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.413 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:06.696 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.545 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:06.829 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.547 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:06.831 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:21.564 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:06.843 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.598 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:25.881 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.599 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:25.883 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.630 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:25.912 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.652 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:25.948 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.653 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:25.932 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.653 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:25.934 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.660 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:25.953 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.688 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:25.971 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:40.690 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:25.987 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.565 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:35.849 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.637 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:35.920 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.676 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:35.960 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.704 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:36.001 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.714 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:36.011 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.719 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:36.002 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.725 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:36.008 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.752 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:36.034 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:35:50.761 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:36.056 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.724 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:46.008 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.729 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:46.012 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.749 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:46.031 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.776 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:46.073 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.778 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:46.062 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.779 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:46.077 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.781 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:46.064 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.796 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:46.094 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:00.799 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:46.082 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.661 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:55.944 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.713 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:55.996 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.714 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:56.009 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.714 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:55.998 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.714 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:56.012 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.718 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:55.999 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.728 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:14:56.012 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.731 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:56.029 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:10.741 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:14:56.021 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.778 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:11.062 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.781 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:11.064 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.801 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:11.085 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.827 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:11.125 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.830 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:11.128 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.832 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:11.116 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.834 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:11.117 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.855 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:11.138 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:25.855 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:11.149 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.497 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:11.779 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.497 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:11.781 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.509 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:11.793 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.511 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:11.808 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.511 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:11.795 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.512 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:11.809 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.512 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:11.796 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.520 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:11.817 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:26.520 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:11.804 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:37.025 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:36:37.003+0000 25a8c640 -1 snap_mapper.add_oid found existing snaps mapped on 76:5ff65171:test-rados-api-smithi029-92481-76::foo:28, removing 2023-08-22T23:36:37.032 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:36:37.018+0000 2d29b640 -1 snap_mapper.add_oid found existing snaps mapped on 76:5ff65171:test-rados-api-smithi029-92481-76::foo:28, removing 2023-08-22T23:36:41.020 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:26.304 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.023 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:26.306 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.058 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:26.355 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.063 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:26.357 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.092 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:26.372 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.092 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:26.375 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.157 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:26.439 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.163 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:26.461 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:41.220 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:26.504 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:42.578 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:36:42.563+0000 2568b640 -1 snap_mapper.add_oid found existing snaps mapped on 76:1b7e42ff:test-rados-api-smithi029-92481-78::foo:29, removing 2023-08-22T23:36:42.591 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:36:42.580+0000 27a90640 -1 snap_mapper.add_oid found existing snaps mapped on 76:1b7e42ff:test-rados-api-smithi029-92481-78::foo:29, removing 2023-08-22T23:36:45.966 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:31.250 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.025 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:31.309 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.036 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:31.333 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.041 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:31.338 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.043 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:31.327 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.057 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:31.354 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.092 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:31.375 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.100 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:31.383 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:46.185 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:31.469 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.934 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:41.218 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.939 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:41.222 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.955 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:41.239 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.988 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:41.272 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.989 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:41.286 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.990 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:41.274 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:55.991 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:41.289 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:56.007 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:41.288 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:36:56.008 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:41.304 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:05.963 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:51.247 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:05.966 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:51.250 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:05.979 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:15:51.263 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:06.012 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:51.295 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:06.013 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:51.310 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:06.015 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:51.298 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:06.016 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:51.314 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:06.029 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:15:51.311 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:06.030 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:51.327 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:08.773 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:37:08.764+0000 24689640 -1 snap_mapper.add_oid found existing snaps mapped on 219:4e952cd0:test-rados-api-smithi029-92481-80::foo:2, removing 2023-08-22T23:37:08.790 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:37:08.776+0000 29293640 -1 snap_mapper.add_oid found existing snaps mapped on 219:4e952cd0:test-rados-api-smithi029-92481-80::foo:2, removing 2023-08-22T23:37:16.060 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:01.344 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.063 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:01.346 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.085 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:01.368 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.104 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:01.388 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.108 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:01.392 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.111 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:01.409 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.115 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:01.413 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.128 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:01.412 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.135 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:01.432 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.973 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:02.270 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.974 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:02.272 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:16.988 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:02.285 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:17.010 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:02.293 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:17.010 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:02.294 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:17.032 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:02.316 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:17.047 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:02.330 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:17.048 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:02.332 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:17.062 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:02.345 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.064 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:11.348 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.068 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:11.352 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.091 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:11.375 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.120 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:11.416 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.120 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:11.417 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.121 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:11.405 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.122 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:11.406 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.140 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:11.438 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:26.152 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:11.436 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.081 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:16.361 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.085 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:16.367 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.103 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:16.387 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.133 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:16.426 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.133 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:16.427 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.145 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:16.441 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.145 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:16.426 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.148 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:16.431 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:31.163 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:16.447 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.257 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:31.541 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.262 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:31.546 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.297 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:31.581 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.318 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:31.615 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.321 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:31.619 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.326 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:31.609 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.327 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:31.611 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.351 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:31.649 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:46.362 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:31.645 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.005 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:32.303 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.006 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:32.304 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.014 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:32.312 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.026 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:32.310 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.027 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:32.311 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.028 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:32.312 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.030 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:32.313 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.036 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:32.319 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.039 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:32.322 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:37:47.767 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (22124 ms) 2023-08-22T23:37:47.767 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-08-22T23:37:47.767 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (24743 ms) 2023-08-22T23:37:47.768 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-08-22T23:37:47.768 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3745 ms) 2023-08-22T23:37:47.768 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-08-22T23:37:47.769 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.769 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5574 ms) 2023-08-22T23:37:47.769 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-08-22T23:37:47.769 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4611 ms) 2023-08-22T23:37:47.770 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-08-22T23:37:47.770 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.770 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4660 ms) 2023-08-22T23:37:47.771 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-08-22T23:37:47.771 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.771 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5603 ms) 2023-08-22T23:37:47.771 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-08-22T23:37:47.772 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.772 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40759 ms) 2023-08-22T23:37:47.772 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-08-22T23:37:47.772 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.773 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18149 ms) 2023-08-22T23:37:47.773 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-08-22T23:37:47.773 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.773 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5572 ms) 2023-08-22T23:37:47.774 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-08-22T23:37:47.774 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.775 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4563 ms) 2023-08-22T23:37:47.775 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-08-22T23:37:47.775 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.776 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26881 ms) 2023-08-22T23:37:47.776 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-08-22T23:37:47.776 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.776 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16907 ms) 2023-08-22T23:37:47.777 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-08-22T23:37:47.777 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.777 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6935 ms) 2023-08-22T23:37:47.777 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-08-22T23:37:47.777 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.778 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5368 ms) 2023-08-22T23:37:47.778 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-08-22T23:37:47.778 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.778 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: waiting for scrubs... 2023-08-22T23:37:47.779 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: done waiting 2023-08-22T23:37:47.779 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25578 ms) 2023-08-22T23:37:47.779 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-08-22T23:37:47.780 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.780 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12582 ms) 2023-08-22T23:37:47.780 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-08-22T23:37:47.780 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.781 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11638 ms) 2023-08-22T23:37:47.781 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-08-22T23:37:47.781 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:37:47.781 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11062 ms) 2023-08-22T23:37:47.782 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-08-22T23:38:01.273 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:46.557 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.279 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:46.563 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.312 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:46.594 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.325 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:46.623 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.328 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:46.612 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.329 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:46.626 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.335 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:46.619 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.358 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:46.656 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:01.364 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:46.648 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.424 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:51.707 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.488 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:51.772 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.508 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:16:51.791 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.553 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:51.850 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.554 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:51.852 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.561 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:51.845 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.569 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:51.866 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.570 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:51.854 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:06.590 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:16:51.874 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.468 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:01.750 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.469 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:01.752 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.489 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:01.772 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.518 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:01.815 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.518 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:01.801 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.519 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:01.802 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.522 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:01.819 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.538 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:01.822 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:16.541 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:01.838 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.474 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:11.758 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.478 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:11.761 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.493 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:11.775 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.526 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:11.822 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.526 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:11.809 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.529 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:11.811 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.533 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:11.827 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.542 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:11.824 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:26.543 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:11.841 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.459 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:21.742 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.511 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:21.795 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.528 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:21.811 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.558 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:21.855 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.563 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:21.860 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.572 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:21.856 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.573 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:21.857 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.575 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:21.872 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:36.594 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:21.878 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.452 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:31.735 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.454 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:31.737 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.471 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:31.755 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.505 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:31.802 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.507 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:31.805 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.523 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:31.806 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.524 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:31.808 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.527 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:31.825 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:46.542 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:31.825 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.472 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:36.755 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.473 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:36.757 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.484 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:36.768 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.523 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:36.807 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.523 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:36.821 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.524 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:36.808 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.525 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:36.823 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.536 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:36.820 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.537 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:36.834 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:38:51.990 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:38:51.978+0000 2568b640 -1 snap_mapper.add_oid found existing snaps mapped on 76:e5ae4096:test-rados-api-smithi029-92481-97::foo:31, removing 2023-08-22T23:38:51.997 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:38:51.991+0000 2b697640 -1 snap_mapper.add_oid found existing snaps mapped on 76:e5ae4096:test-rados-api-smithi029-92481-97::foo:31, removing 2023-08-22T23:39:06.784 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:52.068 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.786 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:52.070 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.816 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:17:52.099 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.846 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:52.130 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.847 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:52.131 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.868 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:52.166 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.875 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:52.172 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.892 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:17:52.176 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:06.908 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:52.205 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.793 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:07.076 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.794 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:07.078 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.834 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:07.118 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.841 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:07.125 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.845 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:07.129 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.852 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:07.150 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.854 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:07.152 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.875 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:07.158 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:21.884 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:07.182 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.786 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:08.084 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.787 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:08.085 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.802 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:08.098 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.805 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:08.089 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.806 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:08.090 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.821 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:08.104 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.920 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:08.204 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.921 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:08.205 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:22.937 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:08.220 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.831 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:12.115 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.833 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:12.117 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.847 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:12.130 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.882 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:12.166 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.883 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:12.167 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.885 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:12.183 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.890 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:12.188 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.899 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:12.183 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:26.904 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:12.202 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.799 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:22.082 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.802 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:22.086 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.829 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:22.112 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.862 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:22.159 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.863 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:22.160 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.864 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:22.147 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.865 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:22.149 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.886 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:22.184 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:36.893 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:22.177 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.841 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:32.125 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.907 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:32.205 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.910 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:32.207 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.912 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:32.196 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.959 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:32.242 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.964 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:32.262 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.982 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:32.266 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:46.983 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:32.267 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:47.028 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:32.311 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.719 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:37.003 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.771 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:37.055 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.790 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:37.074 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.822 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:37.105 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.823 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:37.119 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.824 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:37.122 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.827 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:37.111 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.841 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:37.137 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:39:51.847 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:37.129 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.817 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:52.101 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.819 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:52.103 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.852 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:52.135 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.870 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:52.153 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.871 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:52.155 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.872 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:52.169 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.873 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:52.171 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.901 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:52.199 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:06.916 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:52.200 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.766 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:53.063 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.767 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:53.065 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.777 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:53.074 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.810 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:53.093 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.811 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:53.095 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.823 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:18:53.107 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.828 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:53.111 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.829 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:53.113 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:07.839 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:18:53.123 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.871 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:07.155 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.872 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:07.156 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.904 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:07.187 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.919 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:07.203 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.924 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:07.221 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.924 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:07.208 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.925 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:07.222 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.957 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:07.255 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:21.964 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:07.247 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.880 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:12.163 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.884 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:12.168 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.901 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:12.185 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.933 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:12.217 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.934 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:12.231 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.936 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:12.220 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.937 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:12.234 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.950 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:12.234 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:26.951 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:12.248 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.072 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:22.355 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.075 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:22.359 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.098 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:22.381 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.125 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:22.422 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.126 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:22.424 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.130 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:22.414 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.133 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:22.417 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.145 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:22.442 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:37.153 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:22.437 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:46.957 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:32.241 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:46.958 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:32.242 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:46.971 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:32.255 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:47.007 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:32.305 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:47.009 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:32.307 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:47.015 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:32.299 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:47.017 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:32.301 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:47.025 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:32.323 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:47.030 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:32.313 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.148 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:42.432 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.151 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:42.435 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.174 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:42.457 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.201 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:42.499 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.202 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:42.500 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.203 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:42.487 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.206 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:42.490 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.222 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:42.520 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:40:57.230 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:42.514 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.201 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:57.484 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.206 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:57.490 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.242 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:19:57.526 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.260 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:57.557 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.263 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:57.561 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.278 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:57.562 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.284 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:57.568 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.292 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:57.589 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:12.320 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:19:57.604 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.321 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:04.605 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.324 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:04.608 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.348 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:04.632 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.379 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:04.676 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.381 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:04.665 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.382 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:04.666 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.384 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:04.681 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.406 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:04.690 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:19.406 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:04.704 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.183 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:12.467 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.186 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:12.470 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.203 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:12.487 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.235 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:12.533 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.236 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:12.519 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.238 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:12.536 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.239 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:12.522 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.256 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:12.554 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:27.257 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:12.541 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.201 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:17.485 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.203 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:17.487 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.217 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:17.501 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.245 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:17.529 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.246 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:17.531 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.256 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:17.553 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.257 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:17.555 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.261 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:17.545 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:32.271 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:17.568 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.566 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:27.850 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.570 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:27.854 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.622 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:27.919 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.636 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:27.920 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.638 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:27.922 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.651 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:27.949 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.667 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:27.951 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.677 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:27.975 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:42.690 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:27.974 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:44.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:41:44.186 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11310 ms) 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8475 ms) 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6189 ms) 2023-08-22T23:41:44.187 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26822 ms) 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15225 ms) 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13459 ms) 2023-08-22T23:41:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 00000010 69 73 |is| 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 00000012 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12382 ms) 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: require_osd_release = reef 2023-08-22T23:41:44.189 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7116 ms) 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (816853 ms total) 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1451 ms) 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10773 ms) 2023-08-22T23:41:44.190 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19302 ms) 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31544 ms total) 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-08-22T23:41:44.191 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8011 ms) 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (10104 ms) 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: waiting for scrub... 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: done waiting 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26536 ms) 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-08-22T23:41:44.192 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11807 ms) 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9850 ms) 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9669 ms) 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11553 ms) 2023-08-22T23:41:44.193 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-08-22T23:41:52.323 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:37.607 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.374 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:37.658 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.376 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:37.674 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.377 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:37.661 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.377 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:37.661 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.383 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:37.680 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.402 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:37.686 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.405 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:37.703 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:41:52.411 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:37.695 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.393 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:47.677 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.396 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:47.680 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.414 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:47.698 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.441 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:47.739 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.443 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:47.740 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.448 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:47.731 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.449 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:47.733 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.462 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:47.759 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:02.469 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:47.753 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.414 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:57.698 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.419 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:57.703 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.444 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:20:57.728 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.468 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:57.752 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.470 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:57.768 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.471 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:57.755 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.475 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:57.773 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.499 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:57.797 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:12.505 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:20:57.789 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.513 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:02.797 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.563 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:02.846 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.566 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:02.863 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.569 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:02.867 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.578 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:02.862 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.584 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:02.881 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.613 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:02.896 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.614 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:02.898 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:17.628 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:02.912 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.489 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:17.773 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.492 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:17.776 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.522 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:17.806 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.545 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:17.828 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.546 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:17.830 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.546 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:17.844 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.550 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:17.848 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.575 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:17.859 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:32.579 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:17.876 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.429 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:18.727 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.430 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:18.728 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.438 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:18.736 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.444 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:18.728 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.445 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:18.729 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.458 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:18.742 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.477 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:18.760 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.478 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:18.762 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:33.488 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:18.772 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.596 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:32.880 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.598 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:32.882 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.628 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:32.912 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.647 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:32.945 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.648 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:32.946 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.651 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:32.935 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.652 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:32.936 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.680 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:32.977 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:47.685 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:32.969 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.741 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:38.025 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.747 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:38.031 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.767 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:38.051 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.804 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:38.102 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.811 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:38.109 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.818 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:38.102 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.820 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:38.104 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.829 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:38.127 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:42:52.840 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:38.124 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.652 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:47.935 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.658 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:47.942 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.678 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:47.961 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.705 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:47.989 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.710 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:47.994 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.711 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:48.009 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.715 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:48.012 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.730 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:48.014 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:02.734 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:48.032 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.752 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:58.036 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.758 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:58.042 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.773 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:21:58.057 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.804 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:58.102 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.806 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:58.104 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.807 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:58.091 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.809 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:58.093 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.820 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:58.118 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:12.826 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:21:58.110 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.810 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:03.094 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.814 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:03.098 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.829 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:03.113 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.860 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:03.158 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.861 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:03.159 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.870 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:03.153 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.875 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:03.173 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.876 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:03.160 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:17.890 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:03.174 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:18.935 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:04.233 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:18.937 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:04.221 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:18.939 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:04.237 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:18.970 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:04.254 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:18.985 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:04.269 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:18.987 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:04.271 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:19.008 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:04.291 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:19.009 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:04.294 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:19.012 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:04.310 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:32.967 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:18.251 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:32.972 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:18.256 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.011 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:18.295 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.029 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:18.326 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.031 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:18.329 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.037 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:18.321 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.039 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:18.323 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.061 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:18.359 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:33.074 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:18.358 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.804 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:33.088 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.861 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:33.145 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.883 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:33.166 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.910 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:33.208 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.911 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:33.209 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.925 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:33.209 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.927 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:33.211 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.931 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:33.229 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:47.948 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:33.232 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:52.986 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:38.270 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:52.991 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:38.275 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.011 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:38.295 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.042 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:38.340 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.045 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:38.343 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.051 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:38.334 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.052 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:38.336 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.059 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:38.357 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:43:53.072 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:38.356 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.902 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:48.185 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.908 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:48.191 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.943 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:48.227 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.954 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:48.252 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.956 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:48.240 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.957 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:48.255 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.958 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:48.242 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.979 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:48.277 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:02.982 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:48.266 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.937 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:58.221 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.944 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:58.228 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.960 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:22:58.244 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.984 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:58.267 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.990 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:58.274 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.996 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.294 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:12.997 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.295 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:13.010 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:22:58.294 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:13.014 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.312 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:17.953 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:03.237 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:17.957 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:03.241 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:17.972 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:03.256 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:17.999 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:03.283 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:18.001 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:03.299 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:18.002 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:03.286 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:18.007 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:03.305 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:18.022 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:03.320 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:18.022 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:03.306 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.119 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:18.403 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.124 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:18.408 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.170 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:18.454 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.181 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:18.479 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.184 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:18.482 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.188 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:18.472 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.199 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:18.483 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.229 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:18.526 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:33.236 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:18.520 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.135 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:19.433 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.136 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:19.434 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.146 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:19.444 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.315 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:19.599 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.321 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:19.604 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.334 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:19.618 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.380 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:19.663 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.383 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:19.666 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:34.405 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:19.689 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.393 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:23.677 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.396 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:23.679 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.425 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:23.709 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.465 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:23.762 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.466 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:23.763 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.470 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:23.754 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.471 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:23.755 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.491 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:23.789 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:38.497 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:23.780 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:39.814 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9313 ms) 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14387 ms) 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9987 ms) 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15061 ms) 2023-08-22T23:44:39.815 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-08-22T23:44:39.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9972 ms) 2023-08-22T23:44:39.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-08-22T23:44:39.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8784 ms) 2023-08-22T23:44:39.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-08-22T23:44:39.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: hmm, no HitSet yet 2023-08-22T23:44:39.816 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9277 ms) 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,0 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: first is 1692747779 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,0 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,0 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,0 2023-08-22T23:44:39.817 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,1692747785,1692747786,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,1692747785,1692747786,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,1692747785,1692747786,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,1692747785,1692747786,1692747788,1692747789,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,1692747785,1692747786,1692747788,1692747789,0 2023-08-22T23:44:39.818 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747779,1692747782,1692747783,1692747785,1692747786,1692747788,1692747789,0 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: got ls 1692747783,1692747785,1692747786,1692747788,1692747789,1692747791,1692747792,0 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: first now 1692747783, trimmed 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22648 ms) 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: foo0 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-08-22T23:44:39.819 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18179 ms) 2023-08-22T23:44:39.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-08-22T23:44:39.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (19984 ms) 2023-08-22T23:44:39.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-08-22T23:44:39.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23892 ms) 2023-08-22T23:44:39.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-08-22T23:44:39.820 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5727 ms) 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4405 ms) 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5353 ms) 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4687 ms) 2023-08-22T23:44:39.821 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (269186 ms total) 2023-08-22T23:44:39.822 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: 2023-08-22T23:44:39.822 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-08-22T23:44:39.822 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1272834 ms total) 2023-08-22T23:44:39.822 INFO:tasks.workunit.client.0.smithi029.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-08-22T23:44:39.825 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92230 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92230 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92438 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92438 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.826 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92873 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92873 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92681 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92681 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92554 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92554 2023-08-22T23:44:39.827 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92317 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92317 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92618 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92618 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.828 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92341 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92341 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92830 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92830 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92204 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92204 2023-08-22T23:44:39.829 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92268 2023-08-22T23:44:39.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92268 2023-08-22T23:44:39.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92849 2023-08-22T23:44:39.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92849 2023-08-22T23:44:39.830 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92195 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92195 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92216 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92216 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92470 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92470 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92598 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92598 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92499 2023-08-22T23:44:39.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92499 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92858 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92858 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92819 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92819 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92418 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92418 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92705 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92705 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92188 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92188 2023-08-22T23:44:39.834 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92247 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92247 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92646 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92646 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.835 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92289 2023-08-22T23:44:39.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92289 2023-08-22T23:44:39.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92741 2023-08-22T23:44:39.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92741 2023-08-22T23:44:39.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ for t in "${!pids[@]}" 2023-08-22T23:44:39.836 INFO:tasks.workunit.client.0.smithi029.stderr:+ pid=92792 2023-08-22T23:44:39.837 INFO:tasks.workunit.client.0.smithi029.stderr:+ wait 92792 2023-08-22T23:44:39.838 INFO:tasks.workunit.client.0.smithi029.stderr:+ exit 0 2023-08-22T23:44:39.838 INFO:tasks.workunit.client.0.smithi029.stderr:+ cleanup 2023-08-22T23:44:39.839 INFO:tasks.workunit.client.0.smithi029.stderr:+ pkill -P 92183 2023-08-22T23:44:39.840 INFO:tasks.workunit.client.0.smithi029.stderr:+ true 2023-08-22T23:44:39.840 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-22T23:44:39.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-22T23:44:39.876 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-08-22T23:44:39.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-08-22T23:44:40.215 DEBUG:teuthology.parallel:result is None 2023-08-22T23:44:40.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-08-22T23:44:40.248 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-08-22T23:44:40.248 DEBUG:teuthology.orchestra.run.smithi029:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-08-22T23:44:40.308 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-08-22T23:44:40.309 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-08-22T23:44:40.318 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-08-22T23:44:40.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:44:40.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.463+0000 7fb2f32c5640 10 monclient: get_monmap_and_config 2023-08-22T23:44:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.463+0000 7fb2f32c5640 10 monclient: build_initial_monmap 2023-08-22T23:44:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.463+0000 7fb2f32c5640 10 monclient: monmap: 2023-08-22T23:44:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:40.463771+0000 2023-08-22T23:44:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:40.463771+0000 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:40.465 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 Processor -- start 2023-08-22T23:44:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 -- start start 2023-08-22T23:44:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient: init 2023-08-22T23:44:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-08-22T23:44:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2ec0a7b50 0x7fb2ec0a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb2ec0a7b50 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb2ec0b1380 0x7fb2ec13e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb2ec0b1380 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2ec13ead0 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec0a9e80 con 0x7fb2ec13ead0 2023-08-22T23:44:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec144800 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec0aa0f0 con 0x7fb2ec0b1380 2023-08-22T23:44:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.464+0000 7fb2f32c5640 10 monclient(hunting): authenticate will time out at 2844.714364s 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2ec0a7b50 0x7fb2ec0a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2ec0a7b50 0x7fb2ec0a7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52532/0 (socket says 172.21.15.29:52532) 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 1 -- 172.21.15.29:0/3423092890 learned_addr learned my addr 172.21.15.29:0/3423092890 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 10 monclient(hunting): get_auth_request con 0x7fb2ec0a7b50 auth_method 0 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f1061640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.465+0000 7fb2f0860640 1 --2- 172.21.15.29:0/3423092890 >> v2:172.21.15.52:3300/0 conn(0x7fb2ec0b1380 0x7fb2ec13e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f0860640 10 monclient(hunting): get_auth_request con 0x7fb2ec0b1380 auth_method 0 2023-08-22T23:44:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f0860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f0860640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f0860640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f1061640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f1061640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.466+0000 7fb2f1061640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f0860640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f0860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f0860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f1061640 10 monclient(hunting): handle_auth_done global_id 14727 payload 274 2023-08-22T23:44:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f1061640 1 -- 172.21.15.29:0/3423092890 >> v2:172.21.15.52:3300/0 conn(0x7fb2ec0b1380 msgr2=0x7fb2ec13e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f1061640 1 --2- 172.21.15.29:0/3423092890 >> v2:172.21.15.52:3300/0 conn(0x7fb2ec0b1380 0x7fb2ec13e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f1061640 1 -- 172.21.15.29:0/3423092890 >> v1:172.21.15.52:6789/0 conn(0x7fb2ec13ead0 legacy=0x7fb2ec144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f1061640 10 monclient: _finish_hunting 0 2023-08-22T23:44:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.467+0000 7fb2f1061640 1 monclient: found mon.noname-a 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.468+0000 7fb2f1061640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.468+0000 7fb2f1061640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.468+0000 7fb2f1061640 1 -- 172.21.15.29:0/3423092890 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec1453f0 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.469+0000 7fb2f1061640 1 --2- 172.21.15.29:0/3423092890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec0a7f30 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0b7960 tx=0x7fb2e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e578296ad347d1 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 1 -- 172.21.15.29:0/3423092890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8030860 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: dump: 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: _finish_auth 0 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2f32c5640 5 monclient: authenticate success, global_id 14727 2023-08-22T23:44:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2f32c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 1 -- 172.21.15.29:0/3423092890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e80309c0 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2f32c5640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2f32c5640 4 set_mon_vals no callback set 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 1 -- 172.21.15.29:0/3423092890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8030d50 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.473+0000 7fb2e3fff640 10 monclient: dump: 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:40.480 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:40.480 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:40.480 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 10 monclient: shutdown 2023-08-22T23:44:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3423092890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 msgr2=0x7fb2ec0a7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3423092890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec0a7f30 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0b7960 tx=0x7fb2e802b9a0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3423092890 shutdown_connections 2023-08-22T23:44:40.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3423092890 >> v2:172.21.15.52:3300/0 conn(0x7fb2ec0b1380 0x7fb2ec13e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3423092890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec0a7f30 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3423092890 >> 172.21.15.29:0/3423092890 conn(0x7fb2ec0b6a00 msgr2=0x7fb2ec0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3423092890 shutdown_connections 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3423092890 wait complete. 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 10 monclient: build_initial_monmap 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.474+0000 7fb2f32c5640 10 monclient: monmap: 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:40.485 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 librados: starting msgr at 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 librados: starting objecter 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 Processor -- start 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 -- start start 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 librados: setting wanted keys 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 librados: calling monclient init 2023-08-22T23:44:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient: init 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb2ec0a7b50 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2ec0b1380 0x7fb2ec13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb2ec0b1380 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ec13ead0 0x7fb2ec130de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): picked mon.noname-b con 0x7fb2ec13ead0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec13b170 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec137540 con 0x7fb2ec13ead0 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1376b0 con 0x7fb2ec0b1380 2023-08-22T23:44:40.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.475+0000 7fb2f32c5640 10 monclient(hunting): authenticate will time out at 2844.725445s 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f0860640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2ec0b1380 0x7fb2ec13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52544/0 (socket says 172.21.15.29:52544) 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 1 -- 172.21.15.29:0/3164323446 learned_addr learned my addr 172.21.15.29:0/3164323446 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 10 monclient(hunting): get_auth_request con 0x7fb2ec0a7b50 auth_method 0 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1061640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f0860640 10 monclient(hunting): get_auth_request con 0x7fb2ec0b1380 auth_method 0 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f0860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f0860640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f0860640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.476+0000 7fb2f1862640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ec13ead0 0x7fb2ec130de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1061640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1061640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1061640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): get_auth_request con 0x7fb2ec13ead0 auth_method 0 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f0860640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f0860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:40.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f0860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.477+0000 7fb2f1862640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.479+0000 7fb2f1061640 10 monclient(hunting): handle_auth_done global_id 14730 payload 824 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.479+0000 7fb2f1061640 1 -- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2ec0b1380 msgr2=0x7fb2ec13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2ec0b1380 0x7fb2ec13a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 1 -- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ec13ead0 msgr2=0x7fb2ec130de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ec13ead0 0x7fb2ec130de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 10 monclient: _finish_hunting 0 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 1 monclient: found mon.noname-a 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.480+0000 7fb2f1061640 1 -- 172.21.15.29:0/3164323446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80094d0 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.482+0000 7fb2f1061640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec136910 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000c00 tx=0x7fb2e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.482+0000 7fb2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2ec0a7b50 2023-08-22T23:44:40.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.482+0000 7fb2e1ffb640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:40.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8039c90 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: dump: 2023-08-22T23:44:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: _finish_auth 0 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e8039df0 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8043640 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e1ffb640 10 monclient: dump: 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2f32c5640 5 monclient: authenticate success, global_id 14730 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2f32c5640 10 monclient: _renew_subs 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2f32c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec13ad10 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.485+0000 7fb2e37fe640 4 set_mon_vals no callback set 2023-08-22T23:44:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.486+0000 7fb2f32c5640 10 client.14730.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.486+0000 7fb2f32c5640 10 monclient: _renew_subs 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.486+0000 7fb2f32c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.486+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec131500 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.486+0000 7fb2f32c5640 1 librados: init done 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.487+0000 7fb2f32c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.487+0000 7fb2f32c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.487+0000 7fb2f32c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:40.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.487+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec04dbd0 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.506+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb2e80437a0 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.507+0000 7fb2e1ffb640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb2d006ba60 0x7fb2d006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:40.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.507+0000 7fb2f0860640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb2d006ba60 0x7fb2d006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:40.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.507+0000 7fb2f0860640 10 monclient: get_auth_request con 0x7fb2d006ba60 auth_method 0 2023-08-22T23:44:40.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.508+0000 7fb2e1ffb640 10 client.14730.objecter ms_handle_connect 0x7fb2d006ba60 2023-08-22T23:44:40.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.508+0000 7fb2f0860640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb2d006ba60 0x7fb2d006df20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec137140 tx=0x7fb2dc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fb2e803e070 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 10 client.14730.objecter ms_dispatch 0x7fb2ec000b90 osd_map(918..918 src has 501..918) v4 2023-08-22T23:44:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 3 client.14730.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 3 client.14730.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 20 client.14730.objecter dump_active .. 0 homeless 2023-08-22T23:44:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fb2e8077830 con 0x7fb2ec0a7b50 2023-08-22T23:44:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.760+0000 7fb2e1ffb640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:40.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.939+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2ec144440 con 0x7fb2d006ba60 2023-08-22T23:44:40.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.944+0000 7fb2e1ffb640 1 -- 172.21.15.29:0/3164323446 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151573 (secure 0 0 0) 0x7fb2ec144440 con 0x7fb2d006ba60 2023-08-22T23:44:40.946 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:44:40.947 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:44:40.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 10 librados: watch_flush enter 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 10 librados: watch_flush exit 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 20 client.14730.objecter shutdown clearing up homeless session... 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 10 client.14730.objecter successfully canceled tick 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb2d006ba60 msgr2=0x7fb2d006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb2d006ba60 0x7fb2d006df20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec137140 tx=0x7fb2dc009190 comp rx=0 tx=0).stop 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 10 monclient: shutdown 2023-08-22T23:44:40.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:40.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 msgr2=0x7fb2ec136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:40.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.947+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec136910 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000c00 tx=0x7fb2e8004730 comp rx=0 tx=0).stop 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 shutdown_connections 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fb2d006ba60 0x7fb2d006df20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ec13ead0 0x7fb2ec130de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fb2ec0b1380 0x7fb2ec13a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 --2- 172.21.15.29:0/3164323446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2ec0a7b50 0x7fb2ec136910 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 >> 172.21.15.29:0/3164323446 conn(0x7fb2ec0b6a00 msgr2=0x7fb2ec073190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 shutdown_connections 2023-08-22T23:44:40.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 -- 172.21.15.29:0/3164323446 wait complete. 2023-08-22T23:44:40.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:40.948+0000 7fb2f32c5640 1 librados: shutdown 2023-08-22T23:44:40.959 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2023-08-22T23:44:40.573861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":613436,"kb_used_data":106524,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754361284,"statfs":{"total":773094113280,"available":772465954816,"internally_reserved":0,"allocated":109080576,"data_stored":66978706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":407,"apply_latency_ms":407,"commit_latency_ns":407000000,"apply_latency_ns":407000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088947"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.444846+0000","last_change":"2023-08-22T23:23:40.393453+0000","last_active":"2023-08-22T23:44:34.444846+0000","last_peered":"2023-08-22T23:44:34.444846+0000","last_clean":"2023-08-22T23:44:34.444846+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:34.444846+0000","last_undegraded":"2023-08-22T23:44:34.444846+0000","last_fullsized":"2023-08-22T23:44:34.444846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:46:36.288105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":864,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.394895+0000","last_change":"2023-08-22T23:23:32.307778+0000","last_active":"2023-08-22T23:44:34.394895+0000","last_peered":"2023-08-22T23:44:34.394895+0000","last_clean":"2023-08-22T23:44:34.394895+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:44:34.394895+0000","last_undegraded":"2023-08-22T23:44:34.394895+0000","last_fullsized":"2023-08-22T23:44:34.394895+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:25:17.589038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.880231+0000","last_change":"2023-08-22T23:23:41.258175+0000","last_active":"2023-08-22T23:44:38.880231+0000","last_peered":"2023-08-22T23:44:38.880231+0000","last_clean":"2023-08-22T23:44:38.880231+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:44:38.880231+0000","last_undegraded":"2023-08-22T23:44:38.880231+0000","last_fullsized":"2023-08-22T23:44:38.880231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:29:31.117622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":866,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.293455+0000","last_change":"2023-08-22T23:23:40.333373+0000","last_active":"2023-08-22T23:44:34.293455+0000","last_peered":"2023-08-22T23:44:34.293455+0000","last_clean":"2023-08-22T23:44:34.293455+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:34.293455+0000","last_undegraded":"2023-08-22T23:44:34.293455+0000","last_fullsized":"2023-08-22T23:44:34.293455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:44:22.179832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.602654+0000","last_change":"2023-08-22T23:23:35.301428+0000","last_active":"2023-08-22T23:44:36.602654+0000","last_peered":"2023-08-22T23:44:36.602654+0000","last_clean":"2023-08-22T23:44:36.602654+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:44:36.602654+0000","last_undegraded":"2023-08-22T23:44:36.602654+0000","last_fullsized":"2023-08-22T23:44:36.602654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:04:58.620425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":866,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.296412+0000","last_change":"2023-08-22T23:23:42.559066+0000","last_active":"2023-08-22T23:44:34.296412+0000","last_peered":"2023-08-22T23:44:34.296412+0000","last_clean":"2023-08-22T23:44:34.296412+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:34.296412+0000","last_undegraded":"2023-08-22T23:44:34.296412+0000","last_fullsized":"2023-08-22T23:44:34.296412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:59:53.939044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.556608+0000","last_change":"2023-08-22T23:23:39.799381+0000","last_active":"2023-08-22T23:44:36.556608+0000","last_peered":"2023-08-22T23:44:36.556608+0000","last_clean":"2023-08-22T23:44:36.556608+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:36.556608+0000","last_undegraded":"2023-08-22T23:44:36.556608+0000","last_fullsized":"2023-08-22T23:44:36.556608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:45:47.412302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.335096+0000","last_change":"2023-08-22T23:23:31.473444+0000","last_active":"2023-08-22T23:44:34.335096+0000","last_peered":"2023-08-22T23:44:34.335096+0000","last_clean":"2023-08-22T23:44:34.335096+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:34.335096+0000","last_undegraded":"2023-08-22T23:44:34.335096+0000","last_fullsized":"2023-08-22T23:44:34.335096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:41:55.891550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":861,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.446796+0000","last_change":"2023-08-22T23:23:30.881230+0000","last_active":"2023-08-22T23:44:34.446796+0000","last_peered":"2023-08-22T23:44:34.446796+0000","last_clean":"2023-08-22T23:44:34.446796+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:44:34.446796+0000","last_undegraded":"2023-08-22T23:44:34.446796+0000","last_fullsized":"2023-08-22T23:44:34.446796+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:45:22.079165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":877,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.282867+0000","last_change":"2023-08-22T23:23:57.358686+0000","last_active":"2023-08-22T23:44:34.282867+0000","last_peered":"2023-08-22T23:44:34.282867+0000","last_clean":"2023-08-22T23:44:34.282867+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:34.282867+0000","last_undegraded":"2023-08-22T23:44:34.282867+0000","last_fullsized":"2023-08-22T23:44:34.282867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:14:12.541623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":863,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.228073+0000","last_change":"2023-08-22T23:23:36.215630+0000","last_active":"2023-08-22T23:44:37.228073+0000","last_peered":"2023-08-22T23:44:37.228073+0000","last_clean":"2023-08-22T23:44:37.228073+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:37.228073+0000","last_undegraded":"2023-08-22T23:44:37.228073+0000","last_fullsized":"2023-08-22T23:44:37.228073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:45:18.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.859661+0000","last_change":"2023-08-22T23:23:45.579934+0000","last_active":"2023-08-22T23:44:39.859661+0000","last_peered":"2023-08-22T23:44:39.859661+0000","last_clean":"2023-08-22T23:44:39.859661+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:44:39.859661+0000","last_undegraded":"2023-08-22T23:44:39.859661+0000","last_fullsized":"2023-08-22T23:44:39.859661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:22:30.109602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.896557+0000","last_change":"2023-08-22T23:22:30.241386+0000","last_active":"2023-08-22T23:44:38.896557+0000","last_peered":"2023-08-22T23:44:38.896557+0000","last_clean":"2023-08-22T23:44:38.896557+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:44:38.896557+0000","last_undegraded":"2023-08-22T23:44:38.896557+0000","last_fullsized":"2023-08-22T23:44:38.896557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:44:37.084613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":873,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.199160+0000","last_change":"2023-08-22T23:23:37.170814+0000","last_active":"2023-08-22T23:44:34.199160+0000","last_peered":"2023-08-22T23:44:34.199160+0000","last_clean":"2023-08-22T23:44:34.199160+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:34.199160+0000","last_undegraded":"2023-08-22T23:44:34.199160+0000","last_fullsized":"2023-08-22T23:44:34.199160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:09:15.576464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":878,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.189811+0000","last_change":"2023-08-22T23:22:30.276299+0000","last_active":"2023-08-22T23:44:34.189811+0000","last_peered":"2023-08-22T23:44:34.189811+0000","last_clean":"2023-08-22T23:44:34.189811+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:34.189811+0000","last_undegraded":"2023-08-22T23:44:34.189811+0000","last_fullsized":"2023-08-22T23:44:34.189811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-23T23:36:37.347305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":874,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.205810+0000","last_change":"2023-08-22T23:23:32.260717+0000","last_active":"2023-08-22T23:44:34.205810+0000","last_peered":"2023-08-22T23:44:34.205810+0000","last_clean":"2023-08-22T23:44:34.205810+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:34.205810+0000","last_undegraded":"2023-08-22T23:44:34.205810+0000","last_fullsized":"2023-08-22T23:44:34.205810+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:34:26.225961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.886114+0000","last_change":"2023-08-22T23:22:30.036499+0000","last_active":"2023-08-22T23:44:38.886114+0000","last_peered":"2023-08-22T23:44:38.886114+0000","last_clean":"2023-08-22T23:44:38.886114+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:44:38.886114+0000","last_undegraded":"2023-08-22T23:44:38.886114+0000","last_fullsized":"2023-08-22T23:44:38.886114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:38:38.054037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":857,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.259759+0000","last_change":"2023-08-22T23:23:47.632982+0000","last_active":"2023-08-22T23:44:34.259759+0000","last_peered":"2023-08-22T23:44:34.259759+0000","last_clean":"2023-08-22T23:44:34.259759+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:34.259759+0000","last_undegraded":"2023-08-22T23:44:34.259759+0000","last_fullsized":"2023-08-22T23:44:34.259759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:55:33.947607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":867,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.366073+0000","last_change":"2023-08-22T23:22:30.442903+0000","last_active":"2023-08-22T23:44:34.366073+0000","last_peered":"2023-08-22T23:44:34.366073+0000","last_clean":"2023-08-22T23:44:34.366073+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:34.366073+0000","last_undegraded":"2023-08-22T23:44:34.366073+0000","last_fullsized":"2023-08-22T23:44:34.366073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:53:41.101184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":879,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.315380+0000","last_change":"2023-08-22T23:23:44.752571+0000","last_active":"2023-08-22T23:44:34.315380+0000","last_peered":"2023-08-22T23:44:34.315380+0000","last_clean":"2023-08-22T23:44:34.315380+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:34.315380+0000","last_undegraded":"2023-08-22T23:44:34.315380+0000","last_fullsized":"2023-08-22T23:44:34.315380+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:11:41.512409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":865,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.161011+0000","last_change":"2023-08-22T23:23:43.800737+0000","last_active":"2023-08-22T23:44:37.161011+0000","last_peered":"2023-08-22T23:44:37.161011+0000","last_clean":"2023-08-22T23:44:37.161011+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:37.161011+0000","last_undegraded":"2023-08-22T23:44:37.161011+0000","last_fullsized":"2023-08-22T23:44:37.161011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:51:40.546608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.858703+0000","last_change":"2023-08-22T23:22:30.393248+0000","last_active":"2023-08-22T23:44:38.858703+0000","last_peered":"2023-08-22T23:44:38.858703+0000","last_clean":"2023-08-22T23:44:38.858703+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:38.858703+0000","last_undegraded":"2023-08-22T23:44:38.858703+0000","last_fullsized":"2023-08-22T23:44:38.858703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:16:05.115250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":865,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.332085+0000","last_change":"2023-08-22T23:23:31.308224+0000","last_active":"2023-08-22T23:44:34.332085+0000","last_peered":"2023-08-22T23:44:34.332085+0000","last_clean":"2023-08-22T23:44:34.332085+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:34.332085+0000","last_undegraded":"2023-08-22T23:44:34.332085+0000","last_fullsized":"2023-08-22T23:44:34.332085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:16:42.335154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":877,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.147945+0000","last_change":"2023-08-22T23:22:31.576502+0000","last_active":"2023-08-22T23:44:34.147945+0000","last_peered":"2023-08-22T23:44:34.147945+0000","last_clean":"2023-08-22T23:44:34.147945+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:34.147945+0000","last_undegraded":"2023-08-22T23:44:34.147945+0000","last_fullsized":"2023-08-22T23:44:34.147945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:26:48.005613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"101'7","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.311560+0000","last_change":"2023-08-22T23:23:56.020996+0000","last_active":"2023-08-22T23:44:34.311560+0000","last_peered":"2023-08-22T23:44:34.311560+0000","last_clean":"2023-08-22T23:44:34.311560+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:34.311560+0000","last_undegraded":"2023-08-22T23:44:34.311560+0000","last_fullsized":"2023-08-22T23:44:34.311560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:06:07.542557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.842303+0000","last_change":"2023-08-22T23:22:29.967457+0000","last_active":"2023-08-22T23:44:38.842303+0000","last_peered":"2023-08-22T23:44:38.842303+0000","last_clean":"2023-08-22T23:44:38.842303+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:38.842303+0000","last_undegraded":"2023-08-22T23:44:38.842303+0000","last_fullsized":"2023-08-22T23:44:38.842303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:34:10.033572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":877,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.572507+0000","last_change":"2023-08-22T23:23:34.106212+0000","last_active":"2023-08-22T23:44:36.572507+0000","last_peered":"2023-08-22T23:44:36.572507+0000","last_clean":"2023-08-22T23:44:36.572507+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:36.572507+0000","last_undegraded":"2023-08-22T23:44:36.572507+0000","last_fullsized":"2023-08-22T23:44:36.572507+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:18:39.449835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.870022+0000","last_change":"2023-08-22T23:22:30.329629+0000","last_active":"2023-08-22T23:44:38.870022+0000","last_peered":"2023-08-22T23:44:38.870022+0000","last_clean":"2023-08-22T23:44:38.870022+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:38.870022+0000","last_undegraded":"2023-08-22T23:44:38.870022+0000","last_fullsized":"2023-08-22T23:44:38.870022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:44:32.527595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":979,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.277016+0000","last_change":"2023-08-22T23:22:24.344727+0000","last_active":"2023-08-22T23:44:34.277016+0000","last_peered":"2023-08-22T23:44:34.277016+0000","last_clean":"2023-08-22T23:44:34.277016+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:34.277016+0000","last_undegraded":"2023-08-22T23:44:34.277016+0000","last_fullsized":"2023-08-22T23:44:34.277016+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_clean_scrub_stamp":"2023-08-22T23:22:21.305061+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:48:33.189247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":876,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.521756+0000","last_change":"2023-08-22T23:23:37.130773+0000","last_active":"2023-08-22T23:44:36.521756+0000","last_peered":"2023-08-22T23:44:36.521756+0000","last_clean":"2023-08-22T23:44:36.521756+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:44:36.521756+0000","last_undegraded":"2023-08-22T23:44:36.521756+0000","last_fullsized":"2023-08-22T23:44:36.521756+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:03:26.536335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.847871+0000","last_change":"2023-08-22T23:23:33.761439+0000","last_active":"2023-08-22T23:44:38.847871+0000","last_peered":"2023-08-22T23:44:38.847871+0000","last_clean":"2023-08-22T23:44:38.847871+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:44:38.847871+0000","last_undegraded":"2023-08-22T23:44:38.847871+0000","last_fullsized":"2023-08-22T23:44:38.847871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:21:42.148901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.855176+0000","last_change":"2023-08-22T23:23:33.024596+0000","last_active":"2023-08-22T23:44:38.855176+0000","last_peered":"2023-08-22T23:44:38.855176+0000","last_clean":"2023-08-22T23:44:38.855176+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:44:38.855176+0000","last_undegraded":"2023-08-22T23:44:38.855176+0000","last_fullsized":"2023-08-22T23:44:38.855176+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:01:54.420757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":872,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.157343+0000","last_change":"2023-08-22T23:23:36.066660+0000","last_active":"2023-08-22T23:44:34.157343+0000","last_peered":"2023-08-22T23:44:34.157343+0000","last_clean":"2023-08-22T23:44:34.157343+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:34.157343+0000","last_undegraded":"2023-08-22T23:44:34.157343+0000","last_fullsized":"2023-08-22T23:44:34.157343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:53:08.491032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.189231+0000","last_change":"2023-08-22T23:23:32.334749+0000","last_active":"2023-08-22T23:44:37.189231+0000","last_peered":"2023-08-22T23:44:37.189231+0000","last_clean":"2023-08-22T23:44:37.189231+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:37.189231+0000","last_undegraded":"2023-08-22T23:44:37.189231+0000","last_fullsized":"2023-08-22T23:44:37.189231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:23:58.961035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.883169+0000","last_change":"2023-08-22T23:23:32.954050+0000","last_active":"2023-08-22T23:44:38.883169+0000","last_peered":"2023-08-22T23:44:38.883169+0000","last_clean":"2023-08-22T23:44:38.883169+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:44:38.883169+0000","last_undegraded":"2023-08-22T23:44:38.883169+0000","last_fullsized":"2023-08-22T23:44:38.883169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:37:01.389521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":878,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.285757+0000","last_change":"2023-08-22T23:23:55.264729+0000","last_active":"2023-08-22T23:44:34.285757+0000","last_peered":"2023-08-22T23:44:34.285757+0000","last_clean":"2023-08-22T23:44:34.285757+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:34.285757+0000","last_undegraded":"2023-08-22T23:44:34.285757+0000","last_fullsized":"2023-08-22T23:44:34.285757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:35:59.339638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.255436+0000","last_change":"2023-08-22T23:23:39.717617+0000","last_active":"2023-08-22T23:44:37.255436+0000","last_peered":"2023-08-22T23:44:37.255436+0000","last_clean":"2023-08-22T23:44:37.255436+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:37.255436+0000","last_undegraded":"2023-08-22T23:44:37.255436+0000","last_fullsized":"2023-08-22T23:44:37.255436+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:01:30.780516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":867,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.279154+0000","last_change":"2023-08-22T23:23:43.444588+0000","last_active":"2023-08-22T23:44:34.279154+0000","last_peered":"2023-08-22T23:44:34.279154+0000","last_clean":"2023-08-22T23:44:34.279154+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:34.279154+0000","last_undegraded":"2023-08-22T23:44:34.279154+0000","last_fullsized":"2023-08-22T23:44:34.279154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:36:31.111975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.846100+0000","last_change":"2023-08-22T23:23:45.510476+0000","last_active":"2023-08-22T23:44:38.846100+0000","last_peered":"2023-08-22T23:44:38.846100+0000","last_clean":"2023-08-22T23:44:38.846100+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:44:38.846100+0000","last_undegraded":"2023-08-22T23:44:38.846100+0000","last_fullsized":"2023-08-22T23:44:38.846100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:30:55.850897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.292536+0000","last_change":"2023-08-22T23:23:33.850083+0000","last_active":"2023-08-22T23:44:36.292536+0000","last_peered":"2023-08-22T23:44:36.292536+0000","last_clean":"2023-08-22T23:44:36.292536+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:36.292536+0000","last_undegraded":"2023-08-22T23:44:36.292536+0000","last_fullsized":"2023-08-22T23:44:36.292536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:48:04.458489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.847155+0000","last_change":"2023-08-22T23:23:41.098969+0000","last_active":"2023-08-22T23:44:39.847155+0000","last_peered":"2023-08-22T23:44:39.847155+0000","last_clean":"2023-08-22T23:44:39.847155+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:44:39.847155+0000","last_undegraded":"2023-08-22T23:44:39.847155+0000","last_fullsized":"2023-08-22T23:44:39.847155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:52:09.641904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477004,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66904,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94304936,"statfs":{"total":96636764160,"available":96568254464,"internally_reserved":0,"allocated":13975552,"data_stored":8701029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":13.682,"5min":16.966000000000001,"15min":19.555},"min":{"1min":7.7030000000000003,"5min":4.734,"15min":4.734},"max":{"1min":40.639000000000003,"5min":187.08799999999999,"15min":254.63},"last":7.3460000000000001},{"interface":"front","average":{"1min":17.207999999999998,"5min":23.309999999999999,"15min":22.024999999999999},"min":{"1min":6.1360000000000001,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":47.487000000000002,"5min":330.00099999999998,"15min":330.00099999999998},"last":30.376000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":22.524999999999999,"15min":22.077000000000002},"min":{"1min":6.8159999999999998,"5min":6.8099999999999996,"15min":6.3019999999999996},"max":{"1min":52.658000000000001,"5min":176.803,"15min":176.803},"last":10.028},{"interface":"front","average":{"1min":33.201000000000001,"5min":28.675999999999998,"15min":25.103000000000002},"min":{"1min":8.7319999999999993,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":267.399,"5min":277.048,"15min":277.048},"last":24.405000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.978000000000002,"5min":15.691000000000001,"15min":19.866},"min":{"1min":7.7519999999999998,"5min":6.3860000000000001,"15min":5.766},"max":{"1min":112.148,"5min":112.148,"15min":332.09199999999998},"last":11.475},{"interface":"front","average":{"1min":18.84,"5min":24.292000000000002,"15min":25.629000000000001},"min":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":75.138000000000005,"5min":193.38499999999999,"15min":218.94900000000001},"last":25.276}]},{"osd":4,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":21.919,"5min":22.414999999999999,"15min":25.763999999999999},"min":{"1min":9.3780000000000001,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":107.697,"5min":107.697,"15min":333.517},"last":12.913},{"interface":"front","average":{"1min":21.629000000000001,"5min":28.734000000000002,"15min":29.236000000000001},"min":{"1min":9.7230000000000008,"5min":9.1370000000000005,"15min":8.5779999999999994},"max":{"1min":108.776,"5min":273.65800000000002,"15min":342.31999999999999},"last":29.466999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":22.91,"5min":21.954999999999998,"15min":23.404},"min":{"1min":11.204000000000001,"5min":6.0869999999999997,"15min":6.0869999999999997},"max":{"1min":73.780000000000001,"5min":106.52,"15min":326.71300000000002},"last":26.649000000000001},{"interface":"front","average":{"1min":20.088999999999999,"5min":24.102,"15min":27.399999999999999},"min":{"1min":7.4470000000000001,"5min":7.4470000000000001,"15min":7.2830000000000004},"max":{"1min":99.808000000000007,"5min":196.352,"15min":250.43600000000001},"last":10.839}]},{"osd":6,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.93,"5min":23.151,"15min":26.940999999999999},"min":{"1min":10.054,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":50.091999999999999,"5min":110.843,"15min":267.84800000000001},"last":12.263},{"interface":"front","average":{"1min":24.856000000000002,"5min":30.155999999999999,"15min":29.498999999999999},"min":{"1min":12.632,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":103.387,"5min":198.358,"15min":255.58699999999999},"last":31.390000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":23.925999999999998,"5min":33.804000000000002,"15min":32.649000000000001},"min":{"1min":13.246,"5min":11.016,"15min":10.459},"max":{"1min":76.736999999999995,"5min":274.60899999999998,"15min":274.60899999999998},"last":32.738999999999997},{"interface":"front","average":{"1min":22.082000000000001,"5min":22.986999999999998,"15min":24.119},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":7.2030000000000003},"max":{"1min":98.132999999999996,"5min":114.447,"15min":213.56899999999999},"last":8.4269999999999996}]}]},{"osd":3,"up_from":16,"seq":68719477006,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84584,"kb_used_data":12884,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287256,"statfs":{"total":96636764160,"available":96550150144,"internally_reserved":0,"allocated":13193216,"data_stored":8065623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":17.213000000000001,"5min":15.962999999999999,"15min":21.890000000000001},"min":{"1min":6.5060000000000002,"5min":5.2729999999999997,"15min":5.0359999999999996},"max":{"1min":122.562,"5min":153.98599999999999,"15min":274.45100000000002},"last":11.135},{"interface":"front","average":{"1min":27.960000000000001,"5min":23.204999999999998,"15min":29.393999999999998},"min":{"1min":9.2579999999999991,"5min":7.4489999999999998,"15min":5.2670000000000003},"max":{"1min":215.34800000000001,"5min":215.34800000000001,"15min":356.75599999999997},"last":12.693}]},{"osd":1,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":13.138999999999999,"5min":16.411999999999999,"15min":25.859999999999999},"min":{"1min":5.218,"5min":5.141,"15min":5.141},"max":{"1min":56.652999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":11.866},{"interface":"front","average":{"1min":14.827999999999999,"5min":16.405000000000001,"15min":24.341000000000001},"min":{"1min":6.8529999999999998,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":51.414999999999999,"5min":190.49799999999999,"15min":374.01799999999997},"last":15.992000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":23.094000000000001,"5min":21.440000000000001,"15min":27.952000000000002},"min":{"1min":6.6280000000000001,"5min":6.524,"15min":6.524},"max":{"1min":145.535,"5min":198.649,"15min":353.66000000000003},"last":18.484000000000002},{"interface":"front","average":{"1min":20.449999999999999,"5min":21.263999999999999,"15min":30.486999999999998},"min":{"1min":7.3970000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":108.08,"5min":193.53999999999999,"15min":385.17599999999999},"last":14.114000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.568000000000001,"5min":22.332999999999998,"15min":30.498999999999999},"min":{"1min":9.4250000000000007,"5min":7.2000000000000002,"15min":6.6230000000000002},"max":{"1min":86.454999999999998,"5min":199.42400000000001,"15min":370.84399999999999},"last":21.204999999999998},{"interface":"front","average":{"1min":19.699000000000002,"5min":25.030000000000001,"15min":35.835000000000001},"min":{"1min":8.6219999999999999,"5min":6.726,"15min":6.726},"max":{"1min":64.091999999999999,"5min":231.53,"15min":372.45299999999997},"last":17.559999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":22.189,"15min":32.162999999999997},"min":{"1min":9.7609999999999992,"5min":9.5820000000000007,"15min":8.5419999999999998},"max":{"1min":52.338999999999999,"5min":175.429,"15min":369.33499999999998},"last":25.120999999999999},{"interface":"front","average":{"1min":18.379999999999999,"5min":21.294,"15min":29.463000000000001},"min":{"1min":7.9320000000000004,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":54.829000000000001,"5min":212.09,"15min":289.34300000000002},"last":19.861999999999998}]},{"osd":6,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":18.791,"5min":22.777999999999999,"15min":33.335999999999999},"min":{"1min":8.8979999999999997,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":57.430999999999997,"5min":213.15000000000001,"15min":372.63799999999998},"last":15.125},{"interface":"front","average":{"1min":24.143999999999998,"5min":27.623999999999999,"15min":31.969000000000001},"min":{"1min":14.032,"5min":8.1010000000000009,"15min":7.7830000000000004},"max":{"1min":63.143000000000001,"5min":215.41300000000001,"15min":346.69900000000001},"last":20.538}]},{"osd":7,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.388000000000002,"5min":23.713999999999999,"15min":33.411000000000001},"min":{"1min":10.506,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":58.783999999999999,"5min":206.708,"15min":417.863},"last":22.695},{"interface":"front","average":{"1min":23.991,"5min":28.187999999999999,"15min":31.818999999999999},"min":{"1min":13.135999999999999,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":61.673000000000002,"5min":229.85900000000001,"15min":241.38200000000001},"last":23.719000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66488,"kb_used_data":13220,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305352,"statfs":{"total":96636764160,"available":96568680448,"internally_reserved":0,"allocated":13537280,"data_stored":8257796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":8.4800000000000004,"5min":16.053999999999998,"15min":17.029},"min":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":15.577999999999999,"5min":204.17500000000001,"15min":208.114},"last":21.135000000000002},{"interface":"front","average":{"1min":16.081,"5min":21.925999999999998,"15min":20.050000000000001},"min":{"1min":8.7599999999999998,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":44.661000000000001,"5min":205.833,"15min":205.833},"last":28.797000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":17.501000000000001,"15min":18.324999999999999},"min":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":114.821,"5min":142.42599999999999,"15min":218.06200000000001},"last":7.5439999999999996},{"interface":"front","average":{"1min":24.239999999999998,"5min":22.905000000000001,"15min":21.925999999999998},"min":{"1min":10.121,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":115.809,"5min":145.83000000000001,"15min":230.108},"last":31.318999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:43:54 2023","interfaces":[{"interface":"back","average":{"1min":16.317,"5min":21.879000000000001,"15min":21.068999999999999},"min":{"1min":7.0880000000000001,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":102.65300000000001,"5min":292.49000000000001,"15min":292.49000000000001},"last":8.4860000000000007},{"interface":"front","average":{"1min":20.469999999999999,"5min":28.943999999999999,"15min":25.609000000000002},"min":{"1min":9.5030000000000001,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":107.33799999999999,"5min":304.017,"15min":306.26299999999998},"last":25.576000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":15.987,"5min":19.297999999999998,"15min":21.773},"min":{"1min":8.7170000000000005,"5min":8.3759999999999994,"15min":7.3869999999999996},"max":{"1min":52.591999999999999,"5min":157.559,"15min":187.357},"last":24.66},{"interface":"front","average":{"1min":14.167999999999999,"5min":18.696999999999999,"15min":20.98},"min":{"1min":7.7050000000000001,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":19.834,"5min":146.64599999999999,"15min":206.59100000000001},"last":21.797999999999998}]},{"osd":5,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.920000000000002,"5min":22.468,"15min":24.614000000000001},"min":{"1min":10.135,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":54.234000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":29.536999999999999},{"interface":"front","average":{"1min":22.023,"5min":21.733000000000001,"15min":23.268000000000001},"min":{"1min":8.9309999999999992,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":63.061999999999998,"5min":181.24199999999999,"15min":181.24199999999999},"last":26.881}]},{"osd":6,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":27.670999999999999,"5min":25.062999999999999,"15min":24.218},"min":{"1min":11.868,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":127.512,"5min":189.55000000000001,"15min":189.55000000000001},"last":30.242000000000001},{"interface":"front","average":{"1min":25.172999999999998,"5min":21.625,"15min":24.170000000000002},"min":{"1min":9.8800000000000008,"5min":9.0280000000000005,"15min":8.8819999999999997},"max":{"1min":119.714,"5min":129.928,"15min":187.434},"last":26.263999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":26.472999999999999,"15min":29.672000000000001},"min":{"1min":11.529999999999999,"5min":10.513,"15min":9.9939999999999998},"max":{"1min":55.665999999999997,"5min":195.655,"15min":402.06099999999998},"last":33.277000000000001},{"interface":"front","average":{"1min":17.021000000000001,"5min":21.558,"15min":28.571999999999999},"min":{"1min":8.9019999999999992,"5min":8.6690000000000005,"15min":8.6690000000000005},"max":{"1min":39.817,"5min":158.26499999999999,"15min":427.274},"last":34.084000000000003}]}]},{"osd":0,"up_from":16,"seq":68719477006,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66528,"kb_used_data":13248,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305312,"statfs":{"total":96636764160,"available":96568639488,"internally_reserved":0,"allocated":13565952,"data_stored":8272645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.164,"15min":17.242999999999999},"min":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":28.050000000000001,"5min":125.212,"15min":315.35300000000001},"last":30.100999999999999},{"interface":"front","average":{"1min":11.481999999999999,"5min":19.741,"15min":19.794},"min":{"1min":5.8739999999999997,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":27.388000000000002,"5min":150.68199999999999,"15min":314.375},"last":28.806000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":15.347,"5min":22.693000000000001,"15min":21.102},"min":{"1min":7.5519999999999996,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":42.912999999999997,"5min":133.678,"15min":255.041},"last":10.315},{"interface":"front","average":{"1min":14.930999999999999,"5min":23.047000000000001,"15min":21.428000000000001},"min":{"1min":7.1859999999999999,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":31.559999999999999,"5min":124.077,"15min":266.08600000000001},"last":12.804}]},{"osd":3,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":14.122,"5min":21.184999999999999,"15min":20.628},"min":{"1min":7.6429999999999998,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":77.963999999999999,"5min":140.357,"15min":359.37},"last":11.356999999999999},{"interface":"front","average":{"1min":20.382000000000001,"5min":23.077999999999999,"15min":22.991},"min":{"1min":8.3200000000000003,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":80.066999999999993,"5min":133.27000000000001,"15min":335.35399999999998},"last":12.057}]},{"osd":4,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":17.265999999999998,"5min":21.844999999999999,"15min":22.927},"min":{"1min":9.0359999999999996,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":40.703000000000003,"5min":270.06900000000002,"15min":270.06900000000002},"last":13.423},{"interface":"front","average":{"1min":17.626999999999999,"5min":23.300000000000001,"15min":23.582000000000001},"min":{"1min":7.2880000000000003,"5min":7.1689999999999996,"15min":7.1689999999999996},"max":{"1min":48.825000000000003,"5min":204.54499999999999,"15min":301.78399999999999},"last":15.983000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":18.289999999999999,"5min":24.518000000000001,"15min":25.202999999999999},"min":{"1min":11.141,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":30.463000000000001,"5min":153.78100000000001,"15min":287.80200000000002},"last":263.81599999999997},{"interface":"front","average":{"1min":19.331,"5min":24.251000000000001,"15min":26.376000000000001},"min":{"1min":9.8699999999999992,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":37.241,"5min":130.245,"15min":260.29399999999998},"last":272.935}]},{"osd":6,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":25.445,"5min":28.920999999999999,"15min":25.911000000000001},"min":{"1min":10.363,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":159.46299999999999,"5min":243.15100000000001,"15min":276.935},"last":28.029},{"interface":"front","average":{"1min":21.541,"5min":23.311,"15min":24.834},"min":{"1min":10.583,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":109.886,"5min":241.351,"15min":265.17399999999998},"last":26.350999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":33.301000000000002,"5min":29.783000000000001,"15min":31.905999999999999},"min":{"1min":13.009,"5min":10.702,"15min":7.1760000000000002},"max":{"1min":184.333,"5min":227.852,"15min":770.02200000000005},"last":16.956},{"interface":"front","average":{"1min":31.655000000000001,"5min":27.768000000000001,"15min":31.879999999999999},"min":{"1min":10.598000000000001,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":154.00800000000001,"5min":209.393,"15min":834.71799999999996},"last":17.928000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509710,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66532,"kb_used_data":13264,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305308,"statfs":{"total":96636764160,"available":96568635392,"internally_reserved":0,"allocated":13582336,"data_stored":8302852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":20.427,"5min":17.768000000000001,"15min":18.873999999999999},"min":{"1min":7.3470000000000004,"5min":6.2119999999999997,"15min":4.5570000000000004},"max":{"1min":95.617000000000004,"5min":130.898,"15min":358.06400000000002},"last":6.9870000000000001},{"interface":"front","average":{"1min":24.739999999999998,"5min":24.744,"15min":22.606999999999999},"min":{"1min":10.959,"5min":9.2430000000000003,"15min":4.9699999999999998},"max":{"1min":103.443,"5min":146.55500000000001,"15min":233.333},"last":9.0399999999999991}]},{"osd":1,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.175999999999998,"5min":15.595000000000001,"15min":17.597000000000001},"min":{"1min":6.4939999999999998,"5min":5.1100000000000003,"15min":5.1100000000000003},"max":{"1min":63.090000000000003,"5min":112.752,"15min":222.346},"last":22.611000000000001},{"interface":"front","average":{"1min":20.001999999999999,"5min":19.123000000000001,"15min":20.712},"min":{"1min":8.8339999999999996,"5min":8.0579999999999998,"15min":6.0519999999999996},"max":{"1min":87.179000000000002,"5min":137.86799999999999,"15min":258.90600000000001},"last":8.3179999999999996}]},{"osd":2,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":21.442,"5min":18.684000000000001,"15min":19.425000000000001},"min":{"1min":6.6619999999999999,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":92.921000000000006,"5min":171.38399999999999,"15min":283.65600000000001},"last":28.452999999999999},{"interface":"front","average":{"1min":22.297999999999998,"5min":21.972000000000001,"15min":23.984999999999999},"min":{"1min":9.0169999999999995,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":100.437,"5min":170.02600000000001,"15min":389.19299999999998},"last":33.014000000000003}]},{"osd":3,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":22.475999999999999,"15min":27.465},"min":{"1min":7.6630000000000003,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":208.94800000000001,"5min":214.666,"15min":352.79500000000002},"last":36.924999999999997},{"interface":"front","average":{"1min":25.527000000000001,"5min":23.891999999999999,"15min":24.518000000000001},"min":{"1min":10.247999999999999,"5min":7.4370000000000003,"15min":7.415},"max":{"1min":91.753,"5min":178.21600000000001,"15min":314.69},"last":27.300000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.849,"5min":20.013000000000002,"15min":23.686},"min":{"1min":7.3520000000000003,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":91.307000000000002,"5min":121.392,"15min":316.47500000000002},"last":26.631},{"interface":"front","average":{"1min":23.972999999999999,"5min":27.207999999999998,"15min":30.826000000000001},"min":{"1min":12.016999999999999,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":92.557000000000002,"5min":156.40100000000001,"15min":326.65100000000001},"last":36.167999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":24.760999999999999,"15min":26.306000000000001},"min":{"1min":10.253,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":86.361999999999995,"5min":183.268,"15min":249.37299999999999},"last":31.268999999999998},{"interface":"front","average":{"1min":24.760000000000002,"5min":22.363,"15min":23.788},"min":{"1min":9.9090000000000007,"5min":9.2989999999999995,"15min":7.3849999999999998},"max":{"1min":93.683000000000007,"5min":146.96199999999999,"15min":215.76499999999999},"last":34.747999999999998}]},{"osd":7,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.631,"15min":26.940000000000001},"min":{"1min":10.702,"5min":10.388999999999999,"15min":9.3559999999999999},"max":{"1min":46.847999999999999,"5min":135.65700000000001,"15min":457.08199999999999},"last":40.125},{"interface":"front","average":{"1min":19.600999999999999,"5min":26.585999999999999,"15min":26.331},"min":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":54.695,"5min":247.423,"15min":353.75900000000001},"last":32.137}]}]},{"osd":4,"up_from":15,"seq":64424509709,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88676,"kb_used_data":13264,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283164,"statfs":{"total":96636764160,"available":96545959936,"internally_reserved":0,"allocated":13582336,"data_stored":8298077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":20.602,"5min":22.466999999999999,"15min":22.718},"min":{"1min":4.7880000000000003,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":90.212000000000003,"5min":141.732,"15min":235.06100000000001},"last":10.895},{"interface":"front","average":{"1min":25.978000000000002,"5min":24.094000000000001,"15min":27.666},"min":{"1min":7.5999999999999996,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":112.598,"5min":189.517,"15min":395.15300000000002},"last":8.1780000000000008}]},{"osd":1,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":36.859999999999999,"5min":25.800999999999998,"15min":26.106999999999999},"min":{"1min":6.9509999999999996,"5min":6.7149999999999999,"15min":5.7439999999999998},"max":{"1min":228.94499999999999,"5min":241.87799999999999,"15min":382.04199999999997},"last":23.074999999999999},{"interface":"front","average":{"1min":41.487000000000002,"5min":33.701999999999998,"15min":33.277999999999999},"min":{"1min":11.436999999999999,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":237.38499999999999,"5min":252.441,"15min":382.69},"last":26.588999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":23.855,"5min":24.207999999999998,"15min":25.460999999999999},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":214.36500000000001,"5min":219.37899999999999,"15min":537.76199999999994},"last":47.415999999999997},{"interface":"front","average":{"1min":20.466999999999999,"5min":26.917999999999999,"15min":32.433999999999997},"min":{"1min":6.5999999999999996,"5min":6.5999999999999996,"15min":6.1210000000000004},"max":{"1min":140.50200000000001,"5min":214.50700000000001,"15min":556.82899999999995},"last":37.369}]},{"osd":3,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":24.536999999999999,"5min":24.959,"15min":28.465},"min":{"1min":8.3970000000000002,"5min":7.0449999999999999,"15min":6.6159999999999997},"max":{"1min":160.98400000000001,"5min":161.05199999999999,"15min":354.50400000000002},"last":13.960000000000001},{"interface":"front","average":{"1min":31.042999999999999,"5min":30.719999999999999,"15min":30.981999999999999},"min":{"1min":10.749000000000001,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":226.25700000000001,"5min":226.25700000000001,"15min":339.51499999999999},"last":14.798}]},{"osd":5,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":34.338000000000001,"5min":30.631,"15min":32.762999999999998},"min":{"1min":9.1660000000000004,"5min":7.4160000000000004,"15min":7.2279999999999998},"max":{"1min":139.31899999999999,"5min":238.57300000000001,"15min":380.863},"last":155.17599999999999},{"interface":"front","average":{"1min":26.352,"5min":30.120999999999999,"15min":36.542000000000002},"min":{"1min":7.5999999999999996,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":99.632999999999996,"5min":215.928,"15min":364.09500000000003},"last":158.03899999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":32.222000000000001,"5min":30.013999999999999,"15min":31.196000000000002},"min":{"1min":12.083,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":190.446,"5min":217,"15min":249.29599999999999},"last":70.799000000000007},{"interface":"front","average":{"1min":24.103000000000002,"5min":24.334,"15min":31.428000000000001},"min":{"1min":8.952,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":132.864,"5min":220.87100000000001,"15min":468.887},"last":74.850999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.209,"5min":24.561,"15min":30.809000000000001},"min":{"1min":10.443,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":26.222999999999999,"5min":187.30099999999999,"15min":361.45499999999998},"last":11.305999999999999},{"interface":"front","average":{"1min":19.317,"5min":28.436,"15min":31.718},"min":{"1min":14.016999999999999,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":33.298999999999999,"5min":197.05600000000001,"15min":368.08699999999999},"last":12.84}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88680,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283160,"statfs":{"total":96636764160,"available":96545955840,"internally_reserved":0,"allocated":13598720,"data_stored":8314461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":11.916,"5min":19.649000000000001,"15min":20.076000000000001},"min":{"1min":5.6710000000000003,"5min":5.6710000000000003,"15min":4.3040000000000003},"max":{"1min":38.225999999999999,"5min":307.584,"15min":307.584},"last":10.816000000000001},{"interface":"front","average":{"1min":16.166,"5min":21.260999999999999,"15min":20.367000000000001},"min":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":48.941000000000003,"5min":317.81,"15min":317.81},"last":13.271000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":13.314,"5min":17.096,"15min":25.003},"min":{"1min":6.2629999999999999,"5min":5.077,"15min":5.077},"max":{"1min":22.173999999999999,"5min":159.58799999999999,"15min":319.34500000000003},"last":18.667000000000002},{"interface":"front","average":{"1min":11.484999999999999,"5min":14.808,"15min":20.055},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":26.338000000000001,"5min":160.74600000000001,"15min":195.78999999999999},"last":6.96}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.140000000000001,"5min":17.152999999999999,"15min":23.173999999999999},"min":{"1min":8.6620000000000008,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":47.432000000000002,"5min":148.97800000000001,"15min":772.00400000000002},"last":11.766},{"interface":"front","average":{"1min":17.105,"5min":20.027999999999999,"15min":24.41},"min":{"1min":10.882,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":43.933999999999997,"5min":151.46700000000001,"15min":789.553},"last":8.6929999999999996}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.602,"5min":19.146000000000001,"15min":22.295999999999999},"min":{"1min":7.7370000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":70.665999999999997,"5min":154.44200000000001,"15min":243.33699999999999},"last":9.9440000000000008},{"interface":"front","average":{"1min":18.843,"5min":24.088999999999999,"15min":26.478000000000002},"min":{"1min":6.7880000000000003,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":68.334000000000003,"5min":337.09699999999998,"15min":337.09699999999998},"last":12.621}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":20.622,"5min":21.59,"15min":29.172999999999998},"min":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.0570000000000004},"max":{"1min":109.88800000000001,"5min":111.015,"15min":333.04000000000002},"last":17.716999999999999},{"interface":"front","average":{"1min":18.895,"5min":20.015999999999998,"15min":24.294},"min":{"1min":8.7140000000000004,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":97.224999999999994,"5min":117.06100000000001,"15min":240.084},"last":13.933}]},{"osd":5,"last update":"Tue Aug 22 23:44:05 2023","interfaces":[{"interface":"back","average":{"1min":20.968,"5min":24.206,"15min":30.463999999999999},"min":{"1min":9.9779999999999998,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":60.380000000000003,"5min":247.898,"15min":342.53100000000001},"last":14.548},{"interface":"front","average":{"1min":19.407,"5min":24.077999999999999,"15min":27.297999999999998},"min":{"1min":8.8770000000000007,"5min":8.8770000000000007,"15min":7.4749999999999996},"max":{"1min":49.686,"5min":176.309,"15min":271.47500000000002},"last":20.128}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.318000000000001,"5min":21.977,"15min":25.302},"min":{"1min":8.5459999999999994,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":32.176000000000002,"5min":144.21600000000001,"15min":228.43100000000001},"last":19.504000000000001},{"interface":"front","average":{"1min":17.408000000000001,"5min":23.198,"15min":28.206},"min":{"1min":10.394,"5min":10.090999999999999,"15min":9.2650000000000006},"max":{"1min":24.224,"5min":141.92599999999999,"15min":262.45800000000003},"last":16.774999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542412,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85044,"kb_used_data":13716,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94286796,"statfs":{"total":96636764160,"available":96549679104,"internally_reserved":0,"allocated":14045184,"data_stored":8766223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":87,"apply_latency_ms":87,"commit_latency_ns":87000000,"apply_latency_ns":87000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":12.601000000000001,"5min":16.588000000000001,"15min":20.968},"min":{"1min":6.7779999999999996,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":33.573999999999998,"5min":114.51300000000001,"15min":220.017},"last":9.0960000000000001},{"interface":"front","average":{"1min":18.562000000000001,"5min":22.257999999999999,"15min":27.212},"min":{"1min":10.353999999999999,"5min":6.7919999999999998,"15min":5.9329999999999998},"max":{"1min":32.883000000000003,"5min":115.74299999999999,"15min":264.11900000000003},"last":10.179}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":10.41,"5min":14.409000000000001,"15min":20.576000000000001},"min":{"1min":6.5119999999999996,"5min":6.1289999999999996,"15min":5.9379999999999997},"max":{"1min":16.731000000000002,"5min":102.66200000000001,"15min":343.899},"last":26.073},{"interface":"front","average":{"1min":15.853,"5min":21.071999999999999,"15min":25.216000000000001},"min":{"1min":8.7690000000000001,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":27.713000000000001,"5min":111.873,"15min":270.40499999999997},"last":10.869999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":14.057,"5min":17.219000000000001,"15min":21.003},"min":{"1min":6.4080000000000004,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":28.347000000000001,"5min":116.45,"15min":248.74799999999999},"last":8.125},{"interface":"front","average":{"1min":14.875,"5min":17.582000000000001,"15min":24.184999999999999},"min":{"1min":7.0060000000000002,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":26.632000000000001,"5min":114.843,"15min":256.94400000000002},"last":24.896000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":24.103999999999999,"15min":30.831},"min":{"1min":7.6070000000000002,"5min":6.5739999999999998,"15min":6.5739999999999998},"max":{"1min":50.060000000000002,"5min":231.99799999999999,"15min":257.23500000000001},"last":28.047000000000001},{"interface":"front","average":{"1min":25.657,"5min":23.934999999999999,"15min":24.399000000000001},"min":{"1min":9.8130000000000006,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":62.527999999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":35.063000000000002}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.437999999999999,"5min":25.553000000000001,"15min":28.288},"min":{"1min":9.5950000000000006,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":33.143000000000001,"5min":208.03200000000001,"15min":374.83499999999998},"last":11.807},{"interface":"front","average":{"1min":15.750999999999999,"5min":25.452999999999999,"15min":26.808},"min":{"1min":9.1219999999999999,"5min":7.5389999999999997,"15min":6.3019999999999996},"max":{"1min":22.956,"5min":273.58699999999999,"15min":341.255},"last":30.215}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":21.895,"5min":26.039000000000001,"15min":31.167999999999999},"min":{"1min":9.6560000000000006,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":49.250999999999998,"5min":228.626,"15min":241.26499999999999},"last":35.978000000000002},{"interface":"front","average":{"1min":16.486000000000001,"5min":23.117000000000001,"15min":26.573},"min":{"1min":8.891,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":32.026000000000003,"5min":247.06800000000001,"15min":247.06800000000001},"last":33.091999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":21.434000000000001,"15min":27.062999999999999},"min":{"1min":10.872,"5min":8.2560000000000002,"15min":8.2560000000000002},"max":{"1min":31.568000000000001,"5min":121.524,"15min":317.44600000000003},"last":28.748000000000001},{"interface":"front","average":{"1min":17.907,"5min":25.620000000000001,"15min":28.559999999999999},"min":{"1min":11.589,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":30.829000000000001,"5min":201.81700000000001,"15min":361.93000000000001},"last":31.341000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:44:40.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:44:41.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.077+0000 7f81a6b12640 10 monclient: get_monmap_and_config 2023-08-22T23:44:41.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.077+0000 7f81a6b12640 10 monclient: build_initial_monmap 2023-08-22T23:44:41.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.077+0000 7f81a6b12640 10 monclient: monmap: 2023-08-22T23:44:41.078 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:41.078 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.078 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:41.077734+0000 2023-08-22T23:44:41.078 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:41.077734+0000 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:41.079 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 1 Processor -- start 2023-08-22T23:44:41.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 1 -- start start 2023-08-22T23:44:41.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient: init 2023-08-22T23:44:41.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:41.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81a00a8210 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f81a00b01e0 0x7f81a0144690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient(hunting): picked mon.noname-c con 0x7f81a00b01e0 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81a00b0600 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00a7770 con 0x7f81a00a8210 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a01452e0 con 0x7f81a00b0600 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00a8d30 con 0x7f81a00b01e0 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.078+0000 7f81a6b12640 10 monclient(hunting): authenticate will time out at 2845.328328s 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.079+0000 7f81a48ae640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f81a00a8210 0x7f81a00a85f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48078/0 (socket says 172.21.15.29:48078) 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.079+0000 7f81a48ae640 1 -- 172.21.15.29:0/940120541 learned_addr learned my addr 172.21.15.29:0/940120541 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.079+0000 7f819ffff640 1 --2- 172.21.15.29:0/940120541 >> v2:172.21.15.52:3300/0 conn(0x7f81a00b01e0 0x7f81a0144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): get_auth_request con 0x7f81a00b01e0 auth_method 0 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.080+0000 7f819ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819f7fe640 1 -- 172.21.15.29:0/940120541 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691059805 0 0) 0x7f81a00a7770 con 0x7f81a00a8210 2023-08-22T23:44:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819f7fe640 10 monclient(hunting): _negotiate 2023-08-22T23:44:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819f7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819f7fe640 10 monclient(hunting): my global_id is 14733 2023-08-22T23:44:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819f7fe640 1 -- 172.21.15.29:0/940120541 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c003520 con 0x7f81a00a8210 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 10 monclient(hunting): handle_auth_done global_id 19546 payload 274 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 1 -- 172.21.15.29:0/940120541 >> v1:172.21.15.52:6789/0 conn(0x7f81a00b0600 legacy=0x7f81a0144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 1 -- 172.21.15.29:0/940120541 >> v1:172.21.15.29:6789/0 conn(0x7f81a00a8210 legacy=0x7f81a00a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 1 monclient: found mon.noname-c 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:41.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.082+0000 7f819ffff640 1 -- 172.21.15.29:0/940120541 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a00b2c00 con 0x7f81a00b01e0 2023-08-22T23:44:41.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.087+0000 7f819ffff640 1 --2- 172.21.15.29:0/940120541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a0144690 secure :-1 s=READY pgs=4619 cs=0 l=1 rev1=1 crypto rx=0x7f81880097a0 tx=0x7f818802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754a88e80b9c0b9e server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:41.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 1 -- 172.21.15.29:0/940120541 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188030610 con 0x7f81a00b01e0 2023-08-22T23:44:41.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:41.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: dump: 2023-08-22T23:44:41.097 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.097 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.098 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.098 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.098 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.098 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.099 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.099 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.099 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.100 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: _finish_auth 0 2023-08-22T23:44:41.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:44:41.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:41.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 1 -- 172.21.15.29:0/940120541 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8188030770 con 0x7f81a00b01e0 2023-08-22T23:44:41.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:41.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 1 -- 172.21.15.29:0/940120541 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188030b00 con 0x7f81a00b01e0 2023-08-22T23:44:41.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f81a6b12640 5 monclient: authenticate success, global_id 19546 2023-08-22T23:44:41.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f81a6b12640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:41.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f81a6b12640 4 set_mon_vals no callback set 2023-08-22T23:44:41.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:41.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.091+0000 7f819f7fe640 10 monclient: dump: 2023-08-22T23:44:41.102 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.102 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.103 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.103 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.103 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.103 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.103 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.104 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.104 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.104 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 10 monclient: shutdown 2023-08-22T23:44:41.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 -- 172.21.15.29:0/940120541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 msgr2=0x7f81a0144690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 --2- 172.21.15.29:0/940120541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a0144690 secure :-1 s=READY pgs=4619 cs=0 l=1 rev1=1 crypto rx=0x7f81880097a0 tx=0x7f818802b960 comp rx=0 tx=0).stop 2023-08-22T23:44:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 -- 172.21.15.29:0/940120541 shutdown_connections 2023-08-22T23:44:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 --2- 172.21.15.29:0/940120541 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a0144690 unknown :-1 s=CLOSED pgs=4619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 -- 172.21.15.29:0/940120541 >> 172.21.15.29:0/940120541 conn(0x7f81a00b6a10 msgr2=0x7f81a00b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 -- 172.21.15.29:0/940120541 shutdown_connections 2023-08-22T23:44:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.092+0000 7f81a6b12640 1 -- 172.21.15.29:0/940120541 wait complete. 2023-08-22T23:44:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 10 monclient: build_initial_monmap 2023-08-22T23:44:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 10 monclient: monmap: 2023-08-22T23:44:41.106 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.107 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:41.108 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:41.108 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:41.108 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 1 librados: starting msgr at 2023-08-22T23:44:41.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 1 librados: starting objecter 2023-08-22T23:44:41.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 1 Processor -- start 2023-08-22T23:44:41.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 1 -- start start 2023-08-22T23:44:41.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 1 librados: setting wanted keys 2023-08-22T23:44:41.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 1 librados: calling monclient init 2023-08-22T23:44:41.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.093+0000 7f81a6b12640 10 monclient: init 2023-08-22T23:44:41.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:41.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:44:41.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a00a8210 0x7f81a00abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): picked mon.noname-a con 0x7f81a00a8210 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:41.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a01415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): picked mon.noname-b con 0x7f81a00b01e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81a00b0600 0x7f81a00acba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): picked mon.noname-c con 0x7f81a00b0600 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a00ab480 con 0x7f81a00a8210 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a013d680 con 0x7f81a00b01e0 2023-08-22T23:44:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a013d840 con 0x7f81a00b0600 2023-08-22T23:44:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a6b12640 10 monclient(hunting): authenticate will time out at 2845.343930s 2023-08-22T23:44:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.094+0000 7f81a48ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a00a8210 0x7f81a00abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81a00b0600 0x7f81a00acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a01415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81a00b0600 0x7f81a00acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34152/0 (socket says 172.21.15.29:34152) 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a00a8210 0x7f81a00abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52554/0 (socket says 172.21.15.29:52554) 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 1 -- 172.21.15.29:0/4236588815 learned_addr learned my addr 172.21.15.29:0/4236588815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a01415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:33186/0 (socket says 172.21.15.29:33186) 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): get_auth_request con 0x7f81a00b01e0 auth_method 0 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): get_auth_request con 0x7f81a00a8210 auth_method 0 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): get_auth_request con 0x7f81a00b0600 auth_method 0 2023-08-22T23:44:41.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f819ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a50af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.095+0000 7f81a48ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 10 monclient(hunting): handle_auth_done global_id 19552 payload 824 2023-08-22T23:44:41.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 1 -- 172.21.15.29:0/4236588815 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81a00b0600 msgr2=0x7f81a00acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81a00b0600 0x7f81a00acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 1 -- 172.21.15.29:0/4236588815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a00a8210 msgr2=0x7f81a00abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a00a8210 0x7f81a00abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:41.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 1 monclient: found mon.noname-b 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.098+0000 7f819ffff640 1 -- 172.21.15.29:0/4236588815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188009400 con 0x7f81a00b01e0 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.101+0000 7f819ffff640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a01415a0 secure :-1 s=READY pgs=4620 cs=0 l=1 rev1=1 crypto rx=0x7f8188033000 tx=0x7f8188004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.101+0000 7f819d7fa640 10 client.?.objecter ms_handle_connect 0x7f81a00b01e0 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.101+0000 7f819d7fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:41.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818803e040 con 0x7f81a00b01e0 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 10 monclient: dump: 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.120 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 10 monclient: _finish_auth 0 2023-08-22T23:44:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 10 monclient: _check_auth_tickets 2023-08-22T23:44:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8188039a40 con 0x7f81a00b01e0 2023-08-22T23:44:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.106+0000 7f819d7fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188039df0 con 0x7f81a00b01e0 2023-08-22T23:44:41.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f819d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f819d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:41.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f819d7fa640 10 monclient: dump: 2023-08-22T23:44:41.123 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.123 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.124 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.126 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 5 monclient: authenticate success, global_id 19552 2023-08-22T23:44:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 10 monclient: _renew_subs 2023-08-22T23:44:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f819effd640 4 set_mon_vals no callback set 2023-08-22T23:44:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a00ad0e0 con 0x7f81a00b01e0 2023-08-22T23:44:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 10 client.19552.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 10 monclient: _renew_subs 2023-08-22T23:44:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a01d5df0 con 0x7f81a00b01e0 2023-08-22T23:44:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.107+0000 7f81a6b12640 1 librados: init done 2023-08-22T23:44:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.108+0000 7f81a6b12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.108+0000 7f81a6b12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.108+0000 7f81a6b12640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.108+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a0066cb0 con 0x7f81a00b01e0 2023-08-22T23:44:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.127+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f818802f370 con 0x7f81a00b01e0 2023-08-22T23:44:41.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.128+0000 7f819d7fa640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f818006b9e0 0x7f818006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.128+0000 7f81a48ae640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f818006b9e0 0x7f818006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.128+0000 7f81a48ae640 10 monclient: get_auth_request con 0x7f818006b9e0 auth_method 0 2023-08-22T23:44:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.128+0000 7f819d7fa640 10 client.19552.objecter ms_handle_connect 0x7f818006b9e0 2023-08-22T23:44:41.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.128+0000 7f81a48ae640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f818006b9e0 0x7f818006dea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f81900011e0 tx=0x7f8190007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.366+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f818807dbc0 con 0x7f81a00b01e0 2023-08-22T23:44:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.366+0000 7f819d7fa640 10 client.19552.objecter ms_dispatch 0x7f81a0000b90 osd_map(918..918 src has 501..918) v4 2023-08-22T23:44:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.366+0000 7f819d7fa640 3 client.19552.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.366+0000 7f819d7fa640 3 client.19552.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.367+0000 7f819d7fa640 20 client.19552.objecter dump_active .. 0 homeless 2023-08-22T23:44:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.377+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f81880761f0 con 0x7f81a00b01e0 2023-08-22T23:44:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.377+0000 7f819d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.377+0000 7f819d7fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.552+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81a01d6030 con 0x7f818006b9e0 2023-08-22T23:44:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.557+0000 7f819d7fa640 1 -- 172.21.15.29:0/4236588815 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151573 (secure 0 0 0) 0x7f81a01d6030 con 0x7f818006b9e0 2023-08-22T23:44:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:44:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:44:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.559+0000 7f81a6b12640 10 librados: watch_flush enter 2023-08-22T23:44:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.559+0000 7f81a6b12640 10 librados: watch_flush exit 2023-08-22T23:44:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.559+0000 7f81a6b12640 20 client.19552.objecter shutdown clearing up homeless session... 2023-08-22T23:44:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.559+0000 7f81a6b12640 10 client.19552.objecter successfully canceled tick 2023-08-22T23:44:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f818006b9e0 msgr2=0x7f818006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f818006b9e0 0x7f818006dea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f81900011e0 tx=0x7f8190007d30 comp rx=0 tx=0).stop 2023-08-22T23:44:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 10 monclient: shutdown 2023-08-22T23:44:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 msgr2=0x7f81a01415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a01415a0 secure :-1 s=READY pgs=4620 cs=0 l=1 rev1=1 crypto rx=0x7f8188033000 tx=0x7f8188004730 comp rx=0 tx=0).stop 2023-08-22T23:44:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 shutdown_connections 2023-08-22T23:44:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f818006b9e0 0x7f818006dea0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f81a00b0600 0x7f81a00acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81a00b01e0 0x7f81a01415a0 unknown :-1 s=CLOSED pgs=4620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 --2- 172.21.15.29:0/4236588815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a00a8210 0x7f81a00abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 >> 172.21.15.29:0/4236588815 conn(0x7f81a00b6a10 msgr2=0x7f81a00b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 shutdown_connections 2023-08-22T23:44:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 -- 172.21.15.29:0/4236588815 wait complete. 2023-08-22T23:44:41.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.560+0000 7f81a6b12640 1 librados: shutdown 2023-08-22T23:44:41.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2023-08-22T23:44:40.573861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":613436,"kb_used_data":106524,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754361284,"statfs":{"total":773094113280,"available":772465954816,"internally_reserved":0,"allocated":109080576,"data_stored":66978706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":407,"apply_latency_ms":407,"commit_latency_ns":407000000,"apply_latency_ns":407000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088947"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.444846+0000","last_change":"2023-08-22T23:23:40.393453+0000","last_active":"2023-08-22T23:44:34.444846+0000","last_peered":"2023-08-22T23:44:34.444846+0000","last_clean":"2023-08-22T23:44:34.444846+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:34.444846+0000","last_undegraded":"2023-08-22T23:44:34.444846+0000","last_fullsized":"2023-08-22T23:44:34.444846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:46:36.288105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":864,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.394895+0000","last_change":"2023-08-22T23:23:32.307778+0000","last_active":"2023-08-22T23:44:34.394895+0000","last_peered":"2023-08-22T23:44:34.394895+0000","last_clean":"2023-08-22T23:44:34.394895+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:44:34.394895+0000","last_undegraded":"2023-08-22T23:44:34.394895+0000","last_fullsized":"2023-08-22T23:44:34.394895+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:25:17.589038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.880231+0000","last_change":"2023-08-22T23:23:41.258175+0000","last_active":"2023-08-22T23:44:38.880231+0000","last_peered":"2023-08-22T23:44:38.880231+0000","last_clean":"2023-08-22T23:44:38.880231+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:44:38.880231+0000","last_undegraded":"2023-08-22T23:44:38.880231+0000","last_fullsized":"2023-08-22T23:44:38.880231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:29:31.117622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":866,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.293455+0000","last_change":"2023-08-22T23:23:40.333373+0000","last_active":"2023-08-22T23:44:34.293455+0000","last_peered":"2023-08-22T23:44:34.293455+0000","last_clean":"2023-08-22T23:44:34.293455+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:34.293455+0000","last_undegraded":"2023-08-22T23:44:34.293455+0000","last_fullsized":"2023-08-22T23:44:34.293455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:44:22.179832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.602654+0000","last_change":"2023-08-22T23:23:35.301428+0000","last_active":"2023-08-22T23:44:36.602654+0000","last_peered":"2023-08-22T23:44:36.602654+0000","last_clean":"2023-08-22T23:44:36.602654+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:44:36.602654+0000","last_undegraded":"2023-08-22T23:44:36.602654+0000","last_fullsized":"2023-08-22T23:44:36.602654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:04:58.620425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":866,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.296412+0000","last_change":"2023-08-22T23:23:42.559066+0000","last_active":"2023-08-22T23:44:34.296412+0000","last_peered":"2023-08-22T23:44:34.296412+0000","last_clean":"2023-08-22T23:44:34.296412+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:34.296412+0000","last_undegraded":"2023-08-22T23:44:34.296412+0000","last_fullsized":"2023-08-22T23:44:34.296412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:59:53.939044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.556608+0000","last_change":"2023-08-22T23:23:39.799381+0000","last_active":"2023-08-22T23:44:36.556608+0000","last_peered":"2023-08-22T23:44:36.556608+0000","last_clean":"2023-08-22T23:44:36.556608+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:36.556608+0000","last_undegraded":"2023-08-22T23:44:36.556608+0000","last_fullsized":"2023-08-22T23:44:36.556608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:45:47.412302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.335096+0000","last_change":"2023-08-22T23:23:31.473444+0000","last_active":"2023-08-22T23:44:34.335096+0000","last_peered":"2023-08-22T23:44:34.335096+0000","last_clean":"2023-08-22T23:44:34.335096+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:34.335096+0000","last_undegraded":"2023-08-22T23:44:34.335096+0000","last_fullsized":"2023-08-22T23:44:34.335096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:41:55.891550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":861,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.446796+0000","last_change":"2023-08-22T23:23:30.881230+0000","last_active":"2023-08-22T23:44:34.446796+0000","last_peered":"2023-08-22T23:44:34.446796+0000","last_clean":"2023-08-22T23:44:34.446796+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:44:34.446796+0000","last_undegraded":"2023-08-22T23:44:34.446796+0000","last_fullsized":"2023-08-22T23:44:34.446796+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:45:22.079165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":877,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.282867+0000","last_change":"2023-08-22T23:23:57.358686+0000","last_active":"2023-08-22T23:44:34.282867+0000","last_peered":"2023-08-22T23:44:34.282867+0000","last_clean":"2023-08-22T23:44:34.282867+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:34.282867+0000","last_undegraded":"2023-08-22T23:44:34.282867+0000","last_fullsized":"2023-08-22T23:44:34.282867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:14:12.541623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":863,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.228073+0000","last_change":"2023-08-22T23:23:36.215630+0000","last_active":"2023-08-22T23:44:37.228073+0000","last_peered":"2023-08-22T23:44:37.228073+0000","last_clean":"2023-08-22T23:44:37.228073+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:37.228073+0000","last_undegraded":"2023-08-22T23:44:37.228073+0000","last_fullsized":"2023-08-22T23:44:37.228073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:45:18.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.859661+0000","last_change":"2023-08-22T23:23:45.579934+0000","last_active":"2023-08-22T23:44:39.859661+0000","last_peered":"2023-08-22T23:44:39.859661+0000","last_clean":"2023-08-22T23:44:39.859661+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:44:39.859661+0000","last_undegraded":"2023-08-22T23:44:39.859661+0000","last_fullsized":"2023-08-22T23:44:39.859661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:22:30.109602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.896557+0000","last_change":"2023-08-22T23:22:30.241386+0000","last_active":"2023-08-22T23:44:38.896557+0000","last_peered":"2023-08-22T23:44:38.896557+0000","last_clean":"2023-08-22T23:44:38.896557+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:44:38.896557+0000","last_undegraded":"2023-08-22T23:44:38.896557+0000","last_fullsized":"2023-08-22T23:44:38.896557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:44:37.084613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":873,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.199160+0000","last_change":"2023-08-22T23:23:37.170814+0000","last_active":"2023-08-22T23:44:34.199160+0000","last_peered":"2023-08-22T23:44:34.199160+0000","last_clean":"2023-08-22T23:44:34.199160+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:34.199160+0000","last_undegraded":"2023-08-22T23:44:34.199160+0000","last_fullsized":"2023-08-22T23:44:34.199160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:09:15.576464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":878,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.189811+0000","last_change":"2023-08-22T23:22:30.276299+0000","last_active":"2023-08-22T23:44:34.189811+0000","last_peered":"2023-08-22T23:44:34.189811+0000","last_clean":"2023-08-22T23:44:34.189811+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:34.189811+0000","last_undegraded":"2023-08-22T23:44:34.189811+0000","last_fullsized":"2023-08-22T23:44:34.189811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-23T23:36:37.347305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":874,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.205810+0000","last_change":"2023-08-22T23:23:32.260717+0000","last_active":"2023-08-22T23:44:34.205810+0000","last_peered":"2023-08-22T23:44:34.205810+0000","last_clean":"2023-08-22T23:44:34.205810+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:34.205810+0000","last_undegraded":"2023-08-22T23:44:34.205810+0000","last_fullsized":"2023-08-22T23:44:34.205810+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:34:26.225961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.886114+0000","last_change":"2023-08-22T23:22:30.036499+0000","last_active":"2023-08-22T23:44:38.886114+0000","last_peered":"2023-08-22T23:44:38.886114+0000","last_clean":"2023-08-22T23:44:38.886114+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:44:38.886114+0000","last_undegraded":"2023-08-22T23:44:38.886114+0000","last_fullsized":"2023-08-22T23:44:38.886114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:38:38.054037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":857,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.259759+0000","last_change":"2023-08-22T23:23:47.632982+0000","last_active":"2023-08-22T23:44:34.259759+0000","last_peered":"2023-08-22T23:44:34.259759+0000","last_clean":"2023-08-22T23:44:34.259759+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:34.259759+0000","last_undegraded":"2023-08-22T23:44:34.259759+0000","last_fullsized":"2023-08-22T23:44:34.259759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:55:33.947607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":867,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.366073+0000","last_change":"2023-08-22T23:22:30.442903+0000","last_active":"2023-08-22T23:44:34.366073+0000","last_peered":"2023-08-22T23:44:34.366073+0000","last_clean":"2023-08-22T23:44:34.366073+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:34.366073+0000","last_undegraded":"2023-08-22T23:44:34.366073+0000","last_fullsized":"2023-08-22T23:44:34.366073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:53:41.101184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":879,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.315380+0000","last_change":"2023-08-22T23:23:44.752571+0000","last_active":"2023-08-22T23:44:34.315380+0000","last_peered":"2023-08-22T23:44:34.315380+0000","last_clean":"2023-08-22T23:44:34.315380+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:34.315380+0000","last_undegraded":"2023-08-22T23:44:34.315380+0000","last_fullsized":"2023-08-22T23:44:34.315380+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:11:41.512409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":865,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.161011+0000","last_change":"2023-08-22T23:23:43.800737+0000","last_active":"2023-08-22T23:44:37.161011+0000","last_peered":"2023-08-22T23:44:37.161011+0000","last_clean":"2023-08-22T23:44:37.161011+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:37.161011+0000","last_undegraded":"2023-08-22T23:44:37.161011+0000","last_fullsized":"2023-08-22T23:44:37.161011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:51:40.546608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.858703+0000","last_change":"2023-08-22T23:22:30.393248+0000","last_active":"2023-08-22T23:44:38.858703+0000","last_peered":"2023-08-22T23:44:38.858703+0000","last_clean":"2023-08-22T23:44:38.858703+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:38.858703+0000","last_undegraded":"2023-08-22T23:44:38.858703+0000","last_fullsized":"2023-08-22T23:44:38.858703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:16:05.115250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":865,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.332085+0000","last_change":"2023-08-22T23:23:31.308224+0000","last_active":"2023-08-22T23:44:34.332085+0000","last_peered":"2023-08-22T23:44:34.332085+0000","last_clean":"2023-08-22T23:44:34.332085+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:34.332085+0000","last_undegraded":"2023-08-22T23:44:34.332085+0000","last_fullsized":"2023-08-22T23:44:34.332085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:16:42.335154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":877,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.147945+0000","last_change":"2023-08-22T23:22:31.576502+0000","last_active":"2023-08-22T23:44:34.147945+0000","last_peered":"2023-08-22T23:44:34.147945+0000","last_clean":"2023-08-22T23:44:34.147945+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:34.147945+0000","last_undegraded":"2023-08-22T23:44:34.147945+0000","last_fullsized":"2023-08-22T23:44:34.147945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:26:48.005613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"101'7","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.311560+0000","last_change":"2023-08-22T23:23:56.020996+0000","last_active":"2023-08-22T23:44:34.311560+0000","last_peered":"2023-08-22T23:44:34.311560+0000","last_clean":"2023-08-22T23:44:34.311560+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:34.311560+0000","last_undegraded":"2023-08-22T23:44:34.311560+0000","last_fullsized":"2023-08-22T23:44:34.311560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:06:07.542557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.842303+0000","last_change":"2023-08-22T23:22:29.967457+0000","last_active":"2023-08-22T23:44:38.842303+0000","last_peered":"2023-08-22T23:44:38.842303+0000","last_clean":"2023-08-22T23:44:38.842303+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:38.842303+0000","last_undegraded":"2023-08-22T23:44:38.842303+0000","last_fullsized":"2023-08-22T23:44:38.842303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:34:10.033572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":877,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.572507+0000","last_change":"2023-08-22T23:23:34.106212+0000","last_active":"2023-08-22T23:44:36.572507+0000","last_peered":"2023-08-22T23:44:36.572507+0000","last_clean":"2023-08-22T23:44:36.572507+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:36.572507+0000","last_undegraded":"2023-08-22T23:44:36.572507+0000","last_fullsized":"2023-08-22T23:44:36.572507+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:18:39.449835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.870022+0000","last_change":"2023-08-22T23:22:30.329629+0000","last_active":"2023-08-22T23:44:38.870022+0000","last_peered":"2023-08-22T23:44:38.870022+0000","last_clean":"2023-08-22T23:44:38.870022+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:38.870022+0000","last_undegraded":"2023-08-22T23:44:38.870022+0000","last_fullsized":"2023-08-22T23:44:38.870022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:44:32.527595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":979,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.277016+0000","last_change":"2023-08-22T23:22:24.344727+0000","last_active":"2023-08-22T23:44:34.277016+0000","last_peered":"2023-08-22T23:44:34.277016+0000","last_clean":"2023-08-22T23:44:34.277016+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:34.277016+0000","last_undegraded":"2023-08-22T23:44:34.277016+0000","last_fullsized":"2023-08-22T23:44:34.277016+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_clean_scrub_stamp":"2023-08-22T23:22:21.305061+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:48:33.189247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":876,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.521756+0000","last_change":"2023-08-22T23:23:37.130773+0000","last_active":"2023-08-22T23:44:36.521756+0000","last_peered":"2023-08-22T23:44:36.521756+0000","last_clean":"2023-08-22T23:44:36.521756+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:44:36.521756+0000","last_undegraded":"2023-08-22T23:44:36.521756+0000","last_fullsized":"2023-08-22T23:44:36.521756+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:03:26.536335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.847871+0000","last_change":"2023-08-22T23:23:33.761439+0000","last_active":"2023-08-22T23:44:38.847871+0000","last_peered":"2023-08-22T23:44:38.847871+0000","last_clean":"2023-08-22T23:44:38.847871+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:44:38.847871+0000","last_undegraded":"2023-08-22T23:44:38.847871+0000","last_fullsized":"2023-08-22T23:44:38.847871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:21:42.148901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.855176+0000","last_change":"2023-08-22T23:23:33.024596+0000","last_active":"2023-08-22T23:44:38.855176+0000","last_peered":"2023-08-22T23:44:38.855176+0000","last_clean":"2023-08-22T23:44:38.855176+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:44:38.855176+0000","last_undegraded":"2023-08-22T23:44:38.855176+0000","last_fullsized":"2023-08-22T23:44:38.855176+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:01:54.420757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":872,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.157343+0000","last_change":"2023-08-22T23:23:36.066660+0000","last_active":"2023-08-22T23:44:34.157343+0000","last_peered":"2023-08-22T23:44:34.157343+0000","last_clean":"2023-08-22T23:44:34.157343+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:34.157343+0000","last_undegraded":"2023-08-22T23:44:34.157343+0000","last_fullsized":"2023-08-22T23:44:34.157343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:53:08.491032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.189231+0000","last_change":"2023-08-22T23:23:32.334749+0000","last_active":"2023-08-22T23:44:37.189231+0000","last_peered":"2023-08-22T23:44:37.189231+0000","last_clean":"2023-08-22T23:44:37.189231+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:37.189231+0000","last_undegraded":"2023-08-22T23:44:37.189231+0000","last_fullsized":"2023-08-22T23:44:37.189231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:23:58.961035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.883169+0000","last_change":"2023-08-22T23:23:32.954050+0000","last_active":"2023-08-22T23:44:38.883169+0000","last_peered":"2023-08-22T23:44:38.883169+0000","last_clean":"2023-08-22T23:44:38.883169+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:44:38.883169+0000","last_undegraded":"2023-08-22T23:44:38.883169+0000","last_fullsized":"2023-08-22T23:44:38.883169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:37:01.389521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":878,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.285757+0000","last_change":"2023-08-22T23:23:55.264729+0000","last_active":"2023-08-22T23:44:34.285757+0000","last_peered":"2023-08-22T23:44:34.285757+0000","last_clean":"2023-08-22T23:44:34.285757+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:34.285757+0000","last_undegraded":"2023-08-22T23:44:34.285757+0000","last_fullsized":"2023-08-22T23:44:34.285757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:35:59.339638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.255436+0000","last_change":"2023-08-22T23:23:39.717617+0000","last_active":"2023-08-22T23:44:37.255436+0000","last_peered":"2023-08-22T23:44:37.255436+0000","last_clean":"2023-08-22T23:44:37.255436+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:37.255436+0000","last_undegraded":"2023-08-22T23:44:37.255436+0000","last_fullsized":"2023-08-22T23:44:37.255436+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:01:30.780516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":867,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.279154+0000","last_change":"2023-08-22T23:23:43.444588+0000","last_active":"2023-08-22T23:44:34.279154+0000","last_peered":"2023-08-22T23:44:34.279154+0000","last_clean":"2023-08-22T23:44:34.279154+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:34.279154+0000","last_undegraded":"2023-08-22T23:44:34.279154+0000","last_fullsized":"2023-08-22T23:44:34.279154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:36:31.111975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.846100+0000","last_change":"2023-08-22T23:23:45.510476+0000","last_active":"2023-08-22T23:44:38.846100+0000","last_peered":"2023-08-22T23:44:38.846100+0000","last_clean":"2023-08-22T23:44:38.846100+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:44:38.846100+0000","last_undegraded":"2023-08-22T23:44:38.846100+0000","last_fullsized":"2023-08-22T23:44:38.846100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:30:55.850897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.292536+0000","last_change":"2023-08-22T23:23:33.850083+0000","last_active":"2023-08-22T23:44:36.292536+0000","last_peered":"2023-08-22T23:44:36.292536+0000","last_clean":"2023-08-22T23:44:36.292536+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:36.292536+0000","last_undegraded":"2023-08-22T23:44:36.292536+0000","last_fullsized":"2023-08-22T23:44:36.292536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:48:04.458489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.847155+0000","last_change":"2023-08-22T23:23:41.098969+0000","last_active":"2023-08-22T23:44:39.847155+0000","last_peered":"2023-08-22T23:44:39.847155+0000","last_clean":"2023-08-22T23:44:39.847155+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:44:39.847155+0000","last_undegraded":"2023-08-22T23:44:39.847155+0000","last_fullsized":"2023-08-22T23:44:39.847155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:52:09.641904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477004,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66904,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94304936,"statfs":{"total":96636764160,"available":96568254464,"internally_reserved":0,"allocated":13975552,"data_stored":8701029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":13.682,"5min":16.966000000000001,"15min":19.555},"min":{"1min":7.7030000000000003,"5min":4.734,"15min":4.734},"max":{"1min":40.639000000000003,"5min":187.08799999999999,"15min":254.63},"last":7.3460000000000001},{"interface":"front","average":{"1min":17.207999999999998,"5min":23.309999999999999,"15min":22.024999999999999},"min":{"1min":6.1360000000000001,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":47.487000000000002,"5min":330.00099999999998,"15min":330.00099999999998},"last":30.376000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":22.524999999999999,"15min":22.077000000000002},"min":{"1min":6.8159999999999998,"5min":6.8099999999999996,"15min":6.3019999999999996},"max":{"1min":52.658000000000001,"5min":176.803,"15min":176.803},"last":10.028},{"interface":"front","average":{"1min":33.201000000000001,"5min":28.675999999999998,"15min":25.103000000000002},"min":{"1min":8.7319999999999993,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":267.399,"5min":277.048,"15min":277.048},"last":24.405000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.978000000000002,"5min":15.691000000000001,"15min":19.866},"min":{"1min":7.7519999999999998,"5min":6.3860000000000001,"15min":5.766},"max":{"1min":112.148,"5min":112.148,"15min":332.09199999999998},"last":11.475},{"interface":"front","average":{"1min":18.84,"5min":24.292000000000002,"15min":25.629000000000001},"min":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":75.138000000000005,"5min":193.38499999999999,"15min":218.94900000000001},"last":25.276}]},{"osd":4,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":21.919,"5min":22.414999999999999,"15min":25.763999999999999},"min":{"1min":9.3780000000000001,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":107.697,"5min":107.697,"15min":333.517},"last":12.913},{"interface":"front","average":{"1min":21.629000000000001,"5min":28.734000000000002,"15min":29.236000000000001},"min":{"1min":9.7230000000000008,"5min":9.1370000000000005,"15min":8.5779999999999994},"max":{"1min":108.776,"5min":273.65800000000002,"15min":342.31999999999999},"last":29.466999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":22.91,"5min":21.954999999999998,"15min":23.404},"min":{"1min":11.204000000000001,"5min":6.0869999999999997,"15min":6.0869999999999997},"max":{"1min":73.780000000000001,"5min":106.52,"15min":326.71300000000002},"last":26.649000000000001},{"interface":"front","average":{"1min":20.088999999999999,"5min":24.102,"15min":27.399999999999999},"min":{"1min":7.4470000000000001,"5min":7.4470000000000001,"15min":7.2830000000000004},"max":{"1min":99.808000000000007,"5min":196.352,"15min":250.43600000000001},"last":10.839}]},{"osd":6,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.93,"5min":23.151,"15min":26.940999999999999},"min":{"1min":10.054,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":50.091999999999999,"5min":110.843,"15min":267.84800000000001},"last":12.263},{"interface":"front","average":{"1min":24.856000000000002,"5min":30.155999999999999,"15min":29.498999999999999},"min":{"1min":12.632,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":103.387,"5min":198.358,"15min":255.58699999999999},"last":31.390000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":23.925999999999998,"5min":33.804000000000002,"15min":32.649000000000001},"min":{"1min":13.246,"5min":11.016,"15min":10.459},"max":{"1min":76.736999999999995,"5min":274.60899999999998,"15min":274.60899999999998},"last":32.738999999999997},{"interface":"front","average":{"1min":22.082000000000001,"5min":22.986999999999998,"15min":24.119},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":7.2030000000000003},"max":{"1min":98.132999999999996,"5min":114.447,"15min":213.56899999999999},"last":8.4269999999999996}]}]},{"osd":3,"up_from":16,"seq":68719477006,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84584,"kb_used_data":12884,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287256,"statfs":{"total":96636764160,"available":96550150144,"internally_reserved":0,"allocated":13193216,"data_stored":8065623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":17.213000000000001,"5min":15.962999999999999,"15min":21.890000000000001},"min":{"1min":6.5060000000000002,"5min":5.2729999999999997,"15min":5.0359999999999996},"max":{"1min":122.562,"5min":153.98599999999999,"15min":274.45100000000002},"last":11.135},{"interface":"front","average":{"1min":27.960000000000001,"5min":23.204999999999998,"15min":29.393999999999998},"min":{"1min":9.2579999999999991,"5min":7.4489999999999998,"15min":5.2670000000000003},"max":{"1min":215.34800000000001,"5min":215.34800000000001,"15min":356.75599999999997},"last":12.693}]},{"osd":1,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":13.138999999999999,"5min":16.411999999999999,"15min":25.859999999999999},"min":{"1min":5.218,"5min":5.141,"15min":5.141},"max":{"1min":56.652999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":11.866},{"interface":"front","average":{"1min":14.827999999999999,"5min":16.405000000000001,"15min":24.341000000000001},"min":{"1min":6.8529999999999998,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":51.414999999999999,"5min":190.49799999999999,"15min":374.01799999999997},"last":15.992000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":23.094000000000001,"5min":21.440000000000001,"15min":27.952000000000002},"min":{"1min":6.6280000000000001,"5min":6.524,"15min":6.524},"max":{"1min":145.535,"5min":198.649,"15min":353.66000000000003},"last":18.484000000000002},{"interface":"front","average":{"1min":20.449999999999999,"5min":21.263999999999999,"15min":30.486999999999998},"min":{"1min":7.3970000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":108.08,"5min":193.53999999999999,"15min":385.17599999999999},"last":14.114000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.568000000000001,"5min":22.332999999999998,"15min":30.498999999999999},"min":{"1min":9.4250000000000007,"5min":7.2000000000000002,"15min":6.6230000000000002},"max":{"1min":86.454999999999998,"5min":199.42400000000001,"15min":370.84399999999999},"last":21.204999999999998},{"interface":"front","average":{"1min":19.699000000000002,"5min":25.030000000000001,"15min":35.835000000000001},"min":{"1min":8.6219999999999999,"5min":6.726,"15min":6.726},"max":{"1min":64.091999999999999,"5min":231.53,"15min":372.45299999999997},"last":17.559999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":22.189,"15min":32.162999999999997},"min":{"1min":9.7609999999999992,"5min":9.5820000000000007,"15min":8.5419999999999998},"max":{"1min":52.338999999999999,"5min":175.429,"15min":369.33499999999998},"last":25.120999999999999},{"interface":"front","average":{"1min":18.379999999999999,"5min":21.294,"15min":29.463000000000001},"min":{"1min":7.9320000000000004,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":54.829000000000001,"5min":212.09,"15min":289.34300000000002},"last":19.861999999999998}]},{"osd":6,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":18.791,"5min":22.777999999999999,"15min":33.335999999999999},"min":{"1min":8.8979999999999997,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":57.430999999999997,"5min":213.15000000000001,"15min":372.63799999999998},"last":15.125},{"interface":"front","average":{"1min":24.143999999999998,"5min":27.623999999999999,"15min":31.969000000000001},"min":{"1min":14.032,"5min":8.1010000000000009,"15min":7.7830000000000004},"max":{"1min":63.143000000000001,"5min":215.41300000000001,"15min":346.69900000000001},"last":20.538}]},{"osd":7,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.388000000000002,"5min":23.713999999999999,"15min":33.411000000000001},"min":{"1min":10.506,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":58.783999999999999,"5min":206.708,"15min":417.863},"last":22.695},{"interface":"front","average":{"1min":23.991,"5min":28.187999999999999,"15min":31.818999999999999},"min":{"1min":13.135999999999999,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":61.673000000000002,"5min":229.85900000000001,"15min":241.38200000000001},"last":23.719000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66488,"kb_used_data":13220,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305352,"statfs":{"total":96636764160,"available":96568680448,"internally_reserved":0,"allocated":13537280,"data_stored":8257796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":8.4800000000000004,"5min":16.053999999999998,"15min":17.029},"min":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":15.577999999999999,"5min":204.17500000000001,"15min":208.114},"last":21.135000000000002},{"interface":"front","average":{"1min":16.081,"5min":21.925999999999998,"15min":20.050000000000001},"min":{"1min":8.7599999999999998,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":44.661000000000001,"5min":205.833,"15min":205.833},"last":28.797000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":17.501000000000001,"15min":18.324999999999999},"min":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":114.821,"5min":142.42599999999999,"15min":218.06200000000001},"last":7.5439999999999996},{"interface":"front","average":{"1min":24.239999999999998,"5min":22.905000000000001,"15min":21.925999999999998},"min":{"1min":10.121,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":115.809,"5min":145.83000000000001,"15min":230.108},"last":31.318999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:43:54 2023","interfaces":[{"interface":"back","average":{"1min":16.317,"5min":21.879000000000001,"15min":21.068999999999999},"min":{"1min":7.0880000000000001,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":102.65300000000001,"5min":292.49000000000001,"15min":292.49000000000001},"last":8.4860000000000007},{"interface":"front","average":{"1min":20.469999999999999,"5min":28.943999999999999,"15min":25.609000000000002},"min":{"1min":9.5030000000000001,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":107.33799999999999,"5min":304.017,"15min":306.26299999999998},"last":25.576000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":15.987,"5min":19.297999999999998,"15min":21.773},"min":{"1min":8.7170000000000005,"5min":8.3759999999999994,"15min":7.3869999999999996},"max":{"1min":52.591999999999999,"5min":157.559,"15min":187.357},"last":24.66},{"interface":"front","average":{"1min":14.167999999999999,"5min":18.696999999999999,"15min":20.98},"min":{"1min":7.7050000000000001,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":19.834,"5min":146.64599999999999,"15min":206.59100000000001},"last":21.797999999999998}]},{"osd":5,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.920000000000002,"5min":22.468,"15min":24.614000000000001},"min":{"1min":10.135,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":54.234000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":29.536999999999999},{"interface":"front","average":{"1min":22.023,"5min":21.733000000000001,"15min":23.268000000000001},"min":{"1min":8.9309999999999992,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":63.061999999999998,"5min":181.24199999999999,"15min":181.24199999999999},"last":26.881}]},{"osd":6,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":27.670999999999999,"5min":25.062999999999999,"15min":24.218},"min":{"1min":11.868,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":127.512,"5min":189.55000000000001,"15min":189.55000000000001},"last":30.242000000000001},{"interface":"front","average":{"1min":25.172999999999998,"5min":21.625,"15min":24.170000000000002},"min":{"1min":9.8800000000000008,"5min":9.0280000000000005,"15min":8.8819999999999997},"max":{"1min":119.714,"5min":129.928,"15min":187.434},"last":26.263999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":26.472999999999999,"15min":29.672000000000001},"min":{"1min":11.529999999999999,"5min":10.513,"15min":9.9939999999999998},"max":{"1min":55.665999999999997,"5min":195.655,"15min":402.06099999999998},"last":33.277000000000001},{"interface":"front","average":{"1min":17.021000000000001,"5min":21.558,"15min":28.571999999999999},"min":{"1min":8.9019999999999992,"5min":8.6690000000000005,"15min":8.6690000000000005},"max":{"1min":39.817,"5min":158.26499999999999,"15min":427.274},"last":34.084000000000003}]}]},{"osd":0,"up_from":16,"seq":68719477006,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66528,"kb_used_data":13248,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305312,"statfs":{"total":96636764160,"available":96568639488,"internally_reserved":0,"allocated":13565952,"data_stored":8272645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.164,"15min":17.242999999999999},"min":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":28.050000000000001,"5min":125.212,"15min":315.35300000000001},"last":30.100999999999999},{"interface":"front","average":{"1min":11.481999999999999,"5min":19.741,"15min":19.794},"min":{"1min":5.8739999999999997,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":27.388000000000002,"5min":150.68199999999999,"15min":314.375},"last":28.806000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":15.347,"5min":22.693000000000001,"15min":21.102},"min":{"1min":7.5519999999999996,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":42.912999999999997,"5min":133.678,"15min":255.041},"last":10.315},{"interface":"front","average":{"1min":14.930999999999999,"5min":23.047000000000001,"15min":21.428000000000001},"min":{"1min":7.1859999999999999,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":31.559999999999999,"5min":124.077,"15min":266.08600000000001},"last":12.804}]},{"osd":3,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":14.122,"5min":21.184999999999999,"15min":20.628},"min":{"1min":7.6429999999999998,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":77.963999999999999,"5min":140.357,"15min":359.37},"last":11.356999999999999},{"interface":"front","average":{"1min":20.382000000000001,"5min":23.077999999999999,"15min":22.991},"min":{"1min":8.3200000000000003,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":80.066999999999993,"5min":133.27000000000001,"15min":335.35399999999998},"last":12.057}]},{"osd":4,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":17.265999999999998,"5min":21.844999999999999,"15min":22.927},"min":{"1min":9.0359999999999996,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":40.703000000000003,"5min":270.06900000000002,"15min":270.06900000000002},"last":13.423},{"interface":"front","average":{"1min":17.626999999999999,"5min":23.300000000000001,"15min":23.582000000000001},"min":{"1min":7.2880000000000003,"5min":7.1689999999999996,"15min":7.1689999999999996},"max":{"1min":48.825000000000003,"5min":204.54499999999999,"15min":301.78399999999999},"last":15.983000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":18.289999999999999,"5min":24.518000000000001,"15min":25.202999999999999},"min":{"1min":11.141,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":30.463000000000001,"5min":153.78100000000001,"15min":287.80200000000002},"last":263.81599999999997},{"interface":"front","average":{"1min":19.331,"5min":24.251000000000001,"15min":26.376000000000001},"min":{"1min":9.8699999999999992,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":37.241,"5min":130.245,"15min":260.29399999999998},"last":272.935}]},{"osd":6,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":25.445,"5min":28.920999999999999,"15min":25.911000000000001},"min":{"1min":10.363,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":159.46299999999999,"5min":243.15100000000001,"15min":276.935},"last":28.029},{"interface":"front","average":{"1min":21.541,"5min":23.311,"15min":24.834},"min":{"1min":10.583,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":109.886,"5min":241.351,"15min":265.17399999999998},"last":26.350999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":33.301000000000002,"5min":29.783000000000001,"15min":31.905999999999999},"min":{"1min":13.009,"5min":10.702,"15min":7.1760000000000002},"max":{"1min":184.333,"5min":227.852,"15min":770.02200000000005},"last":16.956},{"interface":"front","average":{"1min":31.655000000000001,"5min":27.768000000000001,"15min":31.879999999999999},"min":{"1min":10.598000000000001,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":154.00800000000001,"5min":209.393,"15min":834.71799999999996},"last":17.928000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509710,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66532,"kb_used_data":13264,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305308,"statfs":{"total":96636764160,"available":96568635392,"internally_reserved":0,"allocated":13582336,"data_stored":8302852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":20.427,"5min":17.768000000000001,"15min":18.873999999999999},"min":{"1min":7.3470000000000004,"5min":6.2119999999999997,"15min":4.5570000000000004},"max":{"1min":95.617000000000004,"5min":130.898,"15min":358.06400000000002},"last":6.9870000000000001},{"interface":"front","average":{"1min":24.739999999999998,"5min":24.744,"15min":22.606999999999999},"min":{"1min":10.959,"5min":9.2430000000000003,"15min":4.9699999999999998},"max":{"1min":103.443,"5min":146.55500000000001,"15min":233.333},"last":9.0399999999999991}]},{"osd":1,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.175999999999998,"5min":15.595000000000001,"15min":17.597000000000001},"min":{"1min":6.4939999999999998,"5min":5.1100000000000003,"15min":5.1100000000000003},"max":{"1min":63.090000000000003,"5min":112.752,"15min":222.346},"last":22.611000000000001},{"interface":"front","average":{"1min":20.001999999999999,"5min":19.123000000000001,"15min":20.712},"min":{"1min":8.8339999999999996,"5min":8.0579999999999998,"15min":6.0519999999999996},"max":{"1min":87.179000000000002,"5min":137.86799999999999,"15min":258.90600000000001},"last":8.3179999999999996}]},{"osd":2,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":21.442,"5min":18.684000000000001,"15min":19.425000000000001},"min":{"1min":6.6619999999999999,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":92.921000000000006,"5min":171.38399999999999,"15min":283.65600000000001},"last":28.452999999999999},{"interface":"front","average":{"1min":22.297999999999998,"5min":21.972000000000001,"15min":23.984999999999999},"min":{"1min":9.0169999999999995,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":100.437,"5min":170.02600000000001,"15min":389.19299999999998},"last":33.014000000000003}]},{"osd":3,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":22.475999999999999,"15min":27.465},"min":{"1min":7.6630000000000003,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":208.94800000000001,"5min":214.666,"15min":352.79500000000002},"last":36.924999999999997},{"interface":"front","average":{"1min":25.527000000000001,"5min":23.891999999999999,"15min":24.518000000000001},"min":{"1min":10.247999999999999,"5min":7.4370000000000003,"15min":7.415},"max":{"1min":91.753,"5min":178.21600000000001,"15min":314.69},"last":27.300000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.849,"5min":20.013000000000002,"15min":23.686},"min":{"1min":7.3520000000000003,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":91.307000000000002,"5min":121.392,"15min":316.47500000000002},"last":26.631},{"interface":"front","average":{"1min":23.972999999999999,"5min":27.207999999999998,"15min":30.826000000000001},"min":{"1min":12.016999999999999,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":92.557000000000002,"5min":156.40100000000001,"15min":326.65100000000001},"last":36.167999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":24.760999999999999,"15min":26.306000000000001},"min":{"1min":10.253,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":86.361999999999995,"5min":183.268,"15min":249.37299999999999},"last":31.268999999999998},{"interface":"front","average":{"1min":24.760000000000002,"5min":22.363,"15min":23.788},"min":{"1min":9.9090000000000007,"5min":9.2989999999999995,"15min":7.3849999999999998},"max":{"1min":93.683000000000007,"5min":146.96199999999999,"15min":215.76499999999999},"last":34.747999999999998}]},{"osd":7,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.631,"15min":26.940000000000001},"min":{"1min":10.702,"5min":10.388999999999999,"15min":9.3559999999999999},"max":{"1min":46.847999999999999,"5min":135.65700000000001,"15min":457.08199999999999},"last":40.125},{"interface":"front","average":{"1min":19.600999999999999,"5min":26.585999999999999,"15min":26.331},"min":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":54.695,"5min":247.423,"15min":353.75900000000001},"last":32.137}]}]},{"osd":4,"up_from":15,"seq":64424509709,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88676,"kb_used_data":13264,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283164,"statfs":{"total":96636764160,"available":96545959936,"internally_reserved":0,"allocated":13582336,"data_stored":8298077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":20.602,"5min":22.466999999999999,"15min":22.718},"min":{"1min":4.7880000000000003,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":90.212000000000003,"5min":141.732,"15min":235.06100000000001},"last":10.895},{"interface":"front","average":{"1min":25.978000000000002,"5min":24.094000000000001,"15min":27.666},"min":{"1min":7.5999999999999996,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":112.598,"5min":189.517,"15min":395.15300000000002},"last":8.1780000000000008}]},{"osd":1,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":36.859999999999999,"5min":25.800999999999998,"15min":26.106999999999999},"min":{"1min":6.9509999999999996,"5min":6.7149999999999999,"15min":5.7439999999999998},"max":{"1min":228.94499999999999,"5min":241.87799999999999,"15min":382.04199999999997},"last":23.074999999999999},{"interface":"front","average":{"1min":41.487000000000002,"5min":33.701999999999998,"15min":33.277999999999999},"min":{"1min":11.436999999999999,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":237.38499999999999,"5min":252.441,"15min":382.69},"last":26.588999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":23.855,"5min":24.207999999999998,"15min":25.460999999999999},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":214.36500000000001,"5min":219.37899999999999,"15min":537.76199999999994},"last":47.415999999999997},{"interface":"front","average":{"1min":20.466999999999999,"5min":26.917999999999999,"15min":32.433999999999997},"min":{"1min":6.5999999999999996,"5min":6.5999999999999996,"15min":6.1210000000000004},"max":{"1min":140.50200000000001,"5min":214.50700000000001,"15min":556.82899999999995},"last":37.369}]},{"osd":3,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":24.536999999999999,"5min":24.959,"15min":28.465},"min":{"1min":8.3970000000000002,"5min":7.0449999999999999,"15min":6.6159999999999997},"max":{"1min":160.98400000000001,"5min":161.05199999999999,"15min":354.50400000000002},"last":13.960000000000001},{"interface":"front","average":{"1min":31.042999999999999,"5min":30.719999999999999,"15min":30.981999999999999},"min":{"1min":10.749000000000001,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":226.25700000000001,"5min":226.25700000000001,"15min":339.51499999999999},"last":14.798}]},{"osd":5,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":34.338000000000001,"5min":30.631,"15min":32.762999999999998},"min":{"1min":9.1660000000000004,"5min":7.4160000000000004,"15min":7.2279999999999998},"max":{"1min":139.31899999999999,"5min":238.57300000000001,"15min":380.863},"last":155.17599999999999},{"interface":"front","average":{"1min":26.352,"5min":30.120999999999999,"15min":36.542000000000002},"min":{"1min":7.5999999999999996,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":99.632999999999996,"5min":215.928,"15min":364.09500000000003},"last":158.03899999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":32.222000000000001,"5min":30.013999999999999,"15min":31.196000000000002},"min":{"1min":12.083,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":190.446,"5min":217,"15min":249.29599999999999},"last":70.799000000000007},{"interface":"front","average":{"1min":24.103000000000002,"5min":24.334,"15min":31.428000000000001},"min":{"1min":8.952,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":132.864,"5min":220.87100000000001,"15min":468.887},"last":74.850999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.209,"5min":24.561,"15min":30.809000000000001},"min":{"1min":10.443,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":26.222999999999999,"5min":187.30099999999999,"15min":361.45499999999998},"last":11.305999999999999},{"interface":"front","average":{"1min":19.317,"5min":28.436,"15min":31.718},"min":{"1min":14.016999999999999,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":33.298999999999999,"5min":197.05600000000001,"15min":368.08699999999999},"last":12.84}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88680,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283160,"statfs":{"total":96636764160,"available":96545955840,"internally_reserved":0,"allocated":13598720,"data_stored":8314461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":11.916,"5min":19.649000000000001,"15min":20.076000000000001},"min":{"1min":5.6710000000000003,"5min":5.6710000000000003,"15min":4.3040000000000003},"max":{"1min":38.225999999999999,"5min":307.584,"15min":307.584},"last":10.816000000000001},{"interface":"front","average":{"1min":16.166,"5min":21.260999999999999,"15min":20.367000000000001},"min":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":48.941000000000003,"5min":317.81,"15min":317.81},"last":13.271000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":13.314,"5min":17.096,"15min":25.003},"min":{"1min":6.2629999999999999,"5min":5.077,"15min":5.077},"max":{"1min":22.173999999999999,"5min":159.58799999999999,"15min":319.34500000000003},"last":18.667000000000002},{"interface":"front","average":{"1min":11.484999999999999,"5min":14.808,"15min":20.055},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":26.338000000000001,"5min":160.74600000000001,"15min":195.78999999999999},"last":6.96}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.140000000000001,"5min":17.152999999999999,"15min":23.173999999999999},"min":{"1min":8.6620000000000008,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":47.432000000000002,"5min":148.97800000000001,"15min":772.00400000000002},"last":11.766},{"interface":"front","average":{"1min":17.105,"5min":20.027999999999999,"15min":24.41},"min":{"1min":10.882,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":43.933999999999997,"5min":151.46700000000001,"15min":789.553},"last":8.6929999999999996}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.602,"5min":19.146000000000001,"15min":22.295999999999999},"min":{"1min":7.7370000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":70.665999999999997,"5min":154.44200000000001,"15min":243.33699999999999},"last":9.9440000000000008},{"interface":"front","average":{"1min":18.843,"5min":24.088999999999999,"15min":26.478000000000002},"min":{"1min":6.7880000000000003,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":68.334000000000003,"5min":337.09699999999998,"15min":337.09699999999998},"last":12.621}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":20.622,"5min":21.59,"15min":29.172999999999998},"min":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.0570000000000004},"max":{"1min":109.88800000000001,"5min":111.015,"15min":333.04000000000002},"last":17.716999999999999},{"interface":"front","average":{"1min":18.895,"5min":20.015999999999998,"15min":24.294},"min":{"1min":8.7140000000000004,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":97.224999999999994,"5min":117.06100000000001,"15min":240.084},"last":13.933}]},{"osd":5,"last update":"Tue Aug 22 23:44:05 2023","interfaces":[{"interface":"back","average":{"1min":20.968,"5min":24.206,"15min":30.463999999999999},"min":{"1min":9.9779999999999998,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":60.380000000000003,"5min":247.898,"15min":342.53100000000001},"last":14.548},{"interface":"front","average":{"1min":19.407,"5min":24.077999999999999,"15min":27.297999999999998},"min":{"1min":8.8770000000000007,"5min":8.8770000000000007,"15min":7.4749999999999996},"max":{"1min":49.686,"5min":176.309,"15min":271.47500000000002},"last":20.128}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.318000000000001,"5min":21.977,"15min":25.302},"min":{"1min":8.5459999999999994,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":32.176000000000002,"5min":144.21600000000001,"15min":228.43100000000001},"last":19.504000000000001},{"interface":"front","average":{"1min":17.408000000000001,"5min":23.198,"15min":28.206},"min":{"1min":10.394,"5min":10.090999999999999,"15min":9.2650000000000006},"max":{"1min":24.224,"5min":141.92599999999999,"15min":262.45800000000003},"last":16.774999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542412,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85044,"kb_used_data":13716,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94286796,"statfs":{"total":96636764160,"available":96549679104,"internally_reserved":0,"allocated":14045184,"data_stored":8766223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":87,"apply_latency_ms":87,"commit_latency_ns":87000000,"apply_latency_ns":87000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":12.601000000000001,"5min":16.588000000000001,"15min":20.968},"min":{"1min":6.7779999999999996,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":33.573999999999998,"5min":114.51300000000001,"15min":220.017},"last":9.0960000000000001},{"interface":"front","average":{"1min":18.562000000000001,"5min":22.257999999999999,"15min":27.212},"min":{"1min":10.353999999999999,"5min":6.7919999999999998,"15min":5.9329999999999998},"max":{"1min":32.883000000000003,"5min":115.74299999999999,"15min":264.11900000000003},"last":10.179}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":10.41,"5min":14.409000000000001,"15min":20.576000000000001},"min":{"1min":6.5119999999999996,"5min":6.1289999999999996,"15min":5.9379999999999997},"max":{"1min":16.731000000000002,"5min":102.66200000000001,"15min":343.899},"last":26.073},{"interface":"front","average":{"1min":15.853,"5min":21.071999999999999,"15min":25.216000000000001},"min":{"1min":8.7690000000000001,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":27.713000000000001,"5min":111.873,"15min":270.40499999999997},"last":10.869999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":14.057,"5min":17.219000000000001,"15min":21.003},"min":{"1min":6.4080000000000004,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":28.347000000000001,"5min":116.45,"15min":248.74799999999999},"last":8.125},{"interface":"front","average":{"1min":14.875,"5min":17.582000000000001,"15min":24.184999999999999},"min":{"1min":7.0060000000000002,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":26.632000000000001,"5min":114.843,"15min":256.94400000000002},"last":24.896000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":24.103999999999999,"15min":30.831},"min":{"1min":7.6070000000000002,"5min":6.5739999999999998,"15min":6.5739999999999998},"max":{"1min":50.060000000000002,"5min":231.99799999999999,"15min":257.23500000000001},"last":28.047000000000001},{"interface":"front","average":{"1min":25.657,"5min":23.934999999999999,"15min":24.399000000000001},"min":{"1min":9.8130000000000006,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":62.527999999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":35.063000000000002}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.437999999999999,"5min":25.553000000000001,"15min":28.288},"min":{"1min":9.5950000000000006,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":33.143000000000001,"5min":208.03200000000001,"15min":374.83499999999998},"last":11.807},{"interface":"front","average":{"1min":15.750999999999999,"5min":25.452999999999999,"15min":26.808},"min":{"1min":9.1219999999999999,"5min":7.5389999999999997,"15min":6.3019999999999996},"max":{"1min":22.956,"5min":273.58699999999999,"15min":341.255},"last":30.215}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":21.895,"5min":26.039000000000001,"15min":31.167999999999999},"min":{"1min":9.6560000000000006,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":49.250999999999998,"5min":228.626,"15min":241.26499999999999},"last":35.978000000000002},{"interface":"front","average":{"1min":16.486000000000001,"5min":23.117000000000001,"15min":26.573},"min":{"1min":8.891,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":32.026000000000003,"5min":247.06800000000001,"15min":247.06800000000001},"last":33.091999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":21.434000000000001,"15min":27.062999999999999},"min":{"1min":10.872,"5min":8.2560000000000002,"15min":8.2560000000000002},"max":{"1min":31.568000000000001,"5min":121.524,"15min":317.44600000000003},"last":28.748000000000001},{"interface":"front","average":{"1min":17.907,"5min":25.620000000000001,"15min":28.559999999999999},"min":{"1min":11.589,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":30.829000000000001,"5min":201.81700000000001,"15min":361.93000000000001},"last":31.341000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:44:41.575 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-08-22T23:44:41.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:44:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.683+0000 7fd7b8628640 10 monclient: get_monmap_and_config 2023-08-22T23:44:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.683+0000 7fd7b8628640 10 monclient: build_initial_monmap 2023-08-22T23:44:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.683+0000 7fd7b8628640 10 monclient: monmap: 2023-08-22T23:44:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:41.683915+0000 2023-08-22T23:44:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:41.683915+0000 2023-08-22T23:44:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:41.687 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 1 Processor -- start 2023-08-22T23:44:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 1 -- start start 2023-08-22T23:44:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 10 monclient: init 2023-08-22T23:44:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b00a7b50 0x7fd7b00a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b00a7b50 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7b00b1380 0x7fd7b013e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b00b1380 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.684+0000 7fd7b8628640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7b013ead0 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b00a9e80 con 0x7fd7b013ead0 2023-08-22T23:44:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b0144800 con 0x7fd7b00b1380 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b00aa0f0 con 0x7fd7b00a7b50 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b8628640 10 monclient(hunting): authenticate will time out at 2845.934509s 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b5bc3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7b00b1380 0x7fd7b013e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b63c4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b00a7b50 0x7fd7b00a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b6bc5640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fd7b013ead0 0x7fd7b0144ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:43750/0 (socket says 172.21.15.29:43750) 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b6bc5640 1 -- 172.21.15.29:0/2795166393 learned_addr learned my addr 172.21.15.29:0/2795166393 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b5bc3640 10 monclient(hunting): get_auth_request con 0x7fd7b00b1380 auth_method 0 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b5bc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b5bc3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b5bc3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b63c4640 10 monclient(hunting): get_auth_request con 0x7fd7b00a7b50 auth_method 0 2023-08-22T23:44:41.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b63c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b63c4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.685+0000 7fd7b63c4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.686+0000 7fd7b5bc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.686+0000 7fd7b5bc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.686+0000 7fd7b5bc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.686+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.686+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.686+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_done global_id 16664 payload 274 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 1 -- 172.21.15.29:0/2795166393 >> v2:172.21.15.29:3300/0 conn(0x7fd7b00b1380 msgr2=0x7fd7b013e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 1 --2- 172.21.15.29:0/2795166393 >> v2:172.21.15.29:3300/0 conn(0x7fd7b00b1380 0x7fd7b013e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 1 -- 172.21.15.29:0/2795166393 >> v1:172.21.15.52:6789/0 conn(0x7fd7b013ead0 legacy=0x7fd7b0144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 10 monclient: _finish_hunting 0 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 1 monclient: found mon.noname-b 2023-08-22T23:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:41.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:41.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b63c4640 1 -- 172.21.15.29:0/2795166393 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b01453f0 con 0x7fd7b00a7b50 2023-08-22T23:44:41.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.688+0000 7fd7b5bc3640 1 --2- 172.21.15.29:0/2795166393 >> v2:172.21.15.29:3300/0 conn(0x7fd7b00b1380 0x7fd7b013e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:41.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.690+0000 7fd7b63c4640 1 --2- 172.21.15.29:0/2795166393 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b00a7b50 0x7fd7b00a7f30 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0097a0 tx=0x7fd7ac02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d3933a3380ed75 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:41.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 1 -- 172.21.15.29:0/2795166393 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac031830 con 0x7fd7b00a7b50 2023-08-22T23:44:41.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:41.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: dump: 2023-08-22T23:44:41.701 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.702 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: _finish_auth 0 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: _check_auth_tickets 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 1 -- 172.21.15.29:0/2795166393 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac031990 con 0x7fd7b00a7b50 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:41.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 1 -- 172.21.15.29:0/2795166393 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac031d20 con 0x7fd7b00a7b50 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b53c2640 10 monclient: dump: 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.704 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b8628640 5 monclient: authenticate success, global_id 16664 2023-08-22T23:44:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b8628640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.693+0000 7fd7b8628640 4 set_mon_vals no callback set 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 10 monclient: shutdown 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 -- 172.21.15.29:0/2795166393 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b00a7b50 msgr2=0x7fd7b00a7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 --2- 172.21.15.29:0/2795166393 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b00a7b50 0x7fd7b00a7f30 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0097a0 tx=0x7fd7ac02b8d0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 -- 172.21.15.29:0/2795166393 shutdown_connections 2023-08-22T23:44:41.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 --2- 172.21.15.29:0/2795166393 >> v2:172.21.15.29:3300/0 conn(0x7fd7b00b1380 0x7fd7b013e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 --2- 172.21.15.29:0/2795166393 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b00a7b50 0x7fd7b00a7f30 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 -- 172.21.15.29:0/2795166393 >> 172.21.15.29:0/2795166393 conn(0x7fd7b00b6a00 msgr2=0x7fd7b00b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 -- 172.21.15.29:0/2795166393 shutdown_connections 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.694+0000 7fd7b8628640 1 -- 172.21.15.29:0/2795166393 wait complete. 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 10 monclient: build_initial_monmap 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 10 monclient: monmap: 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:41.707 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:41.708 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 1 librados: starting msgr at 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 1 librados: starting objecter 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 1 Processor -- start 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 1 -- start start 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 1 librados: setting wanted keys 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 1 librados: calling monclient init 2023-08-22T23:44:41.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.695+0000 7fd7b8628640 10 monclient: init 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 0x7fd7b0143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b00a7b50 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd7b00b1380 0x7fd7b0134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b00b1380 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b013ead0 0x7fd7b01d9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b013ead0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b013a7d0 con 0x7fd7b00a7b50 2023-08-22T23:44:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b013a480 con 0x7fd7b00b1380 2023-08-22T23:44:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b0143dc0 con 0x7fd7b013ead0 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b8628640 10 monclient(hunting): authenticate will time out at 2845.945834s 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b6bc5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b013ead0 0x7fd7b01d9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.696+0000 7fd7b63c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 0x7fd7b0143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b5bc3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd7b00b1380 0x7fd7b0134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 0x7fd7b0143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52580/0 (socket says 172.21.15.29:52580) 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 1 -- 172.21.15.29:0/3660767928 learned_addr learned my addr 172.21.15.29:0/3660767928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): get_auth_request con 0x7fd7b00a7b50 auth_method 0 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): get_auth_request con 0x7fd7b013ead0 auth_method 0 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b5bc3640 10 monclient(hunting): get_auth_request con 0x7fd7b00b1380 auth_method 0 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b5bc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b5bc3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b5bc3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.697+0000 7fd7b6bc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.698+0000 7fd7b5bc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.698+0000 7fd7b5bc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.698+0000 7fd7b5bc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 10 monclient(hunting): handle_auth_done global_id 14742 payload 824 2023-08-22T23:44:41.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 1 -- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b013ead0 msgr2=0x7fd7b01d9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b013ead0 0x7fd7b01d9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 1 -- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd7b00b1380 msgr2=0x7fd7b0134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd7b00b1380 0x7fd7b0134e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 10 monclient: _finish_hunting 0 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 1 monclient: found mon.noname-a 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:41.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:41.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b63c4640 1 -- 172.21.15.29:0/3660767928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0009400 con 0x7fd7b00a7b50 2023-08-22T23:44:41.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.700+0000 7fd7b6bc5640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b013ead0 0x7fd7b01d9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.702+0000 7fd7b63c4640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 0x7fd7b0143720 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac02bee0 tx=0x7fd7ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.702+0000 7fd7a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd7b00a7b50 2023-08-22T23:44:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.702+0000 7fd7a6ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac03a6f0 con 0x7fd7b00a7b50 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: dump: 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac03a870 con 0x7fd7b00a7b50 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 5 monclient: authenticate success, global_id 14742 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 10 monclient: _renew_subs 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac009400 con 0x7fd7b00a7b50 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:41.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac03ac20 con 0x7fd7b00a7b50 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7a6ffd640 10 monclient: dump: 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.722 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b4bc1640 4 set_mon_vals no callback set 2023-08-22T23:44:41.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 10 client.14742.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 10 monclient: _renew_subs 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b013abf0 con 0x7fd7b00a7b50 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.706+0000 7fd7b8628640 1 librados: init done 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.707+0000 7fd7b8628640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.707+0000 7fd7b8628640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:41.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.707+0000 7fd7b8628640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:41.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.707+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b013a910 con 0x7fd7b00a7b50 2023-08-22T23:44:41.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.726+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd7ac04c3b0 con 0x7fd7b00a7b50 2023-08-22T23:44:41.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.727+0000 7fd7a6ffd640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd79406ba30 0x7fd79406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:41.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.727+0000 7fd7b5bc3640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd79406ba30 0x7fd79406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:41.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.727+0000 7fd7b5bc3640 10 monclient: get_auth_request con 0x7fd79406ba30 auth_method 0 2023-08-22T23:44:41.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.728+0000 7fd7b5bc3640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd79406ba30 0x7fd79406def0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd7b01da2f0 tx=0x7fd7a0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:41.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.728+0000 7fd7a6ffd640 10 client.14742.objecter ms_handle_connect 0x7fd79406ba30 2023-08-22T23:44:41.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fd7ac07d640 con 0x7fd7b00a7b50 2023-08-22T23:44:41.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 10 client.14742.objecter ms_dispatch 0x7fd7b0000b90 osd_map(918..918 src has 501..918) v4 2023-08-22T23:44:41.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 3 client.14742.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:41.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 3 client.14742.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:41.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 20 client.14742.objecter dump_active .. 0 homeless 2023-08-22T23:44:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fd7ac075c70 con 0x7fd7b00a7b50 2023-08-22T23:44:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:41.977+0000 7fd7a6ffd640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.174+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7b014b9f0 con 0x7fd79406ba30 2023-08-22T23:44:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.179+0000 7fd7a6ffd640 1 -- 172.21.15.29:0/3660767928 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151573 (secure 0 0 0) 0x7fd7a0009940 con 0x7fd79406ba30 2023-08-22T23:44:42.181 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:44:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:44:42.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 10 librados: watch_flush enter 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 10 librados: watch_flush exit 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 20 client.14742.objecter shutdown clearing up homeless session... 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 10 client.14742.objecter successfully canceled tick 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd79406ba30 msgr2=0x7fd79406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd79406ba30 0x7fd79406def0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd7b01da2f0 tx=0x7fd7a0009190 comp rx=0 tx=0).stop 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 10 monclient: shutdown 2023-08-22T23:44:42.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 msgr2=0x7fd7b0143720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 0x7fd7b0143720 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac02bee0 tx=0x7fd7ac0046c0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 shutdown_connections 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd79406ba30 0x7fd79406def0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd7b013ead0 0x7fd7b01d9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd7b00b1380 0x7fd7b0134e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 --2- 172.21.15.29:0/3660767928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7b00a7b50 0x7fd7b0143720 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 >> 172.21.15.29:0/3660767928 conn(0x7fd7b00b6a00 msgr2=0x7fd7b0073190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:42.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 shutdown_connections 2023-08-22T23:44:42.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 -- 172.21.15.29:0/3660767928 wait complete. 2023-08-22T23:44:42.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.182+0000 7fd7b8628640 1 librados: shutdown 2023-08-22T23:44:42.193 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2023-08-22T23:44:40.573861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":613436,"kb_used_data":106524,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754361284,"statfs":{"total":773094113280,"available":772465954816,"internally_reserved":0,"allocated":109080576,"data_stored":66978706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":407,"apply_latency_ms":407,"commit_latency_ns":407000000,"apply_latency_ns":407000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088947"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.444846+0000","last_change":"2023-08-22T23:23:40.393453+0000","last_active":"2023-08-22T23:44:34.444846+0000","last_peered":"2023-08-22T23:44:34.444846+0000","last_clean":"2023-08-22T23:44:34.444846+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:34.444846+0000","last_undegraded":"2023-08-22T23:44:34.444846+0000","last_fullsized":"2023-08-22T23:44:34.444846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:46:36.288105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":864,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.394895+0000","last_change":"2023-08-22T23:23:32.307778+0000","last_active":"2023-08-22T23:44:34.394895+0000","last_peered":"2023-08-22T23:44:34.394895+0000","last_clean":"2023-08-22T23:44:34.394895+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:44:34.394895+0000","last_undegraded":"2023-08-22T23:44:34.394895+0000","last_fullsized":"2023-08-22T23:44:34.394895+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:25:17.589038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.880231+0000","last_change":"2023-08-22T23:23:41.258175+0000","last_active":"2023-08-22T23:44:38.880231+0000","last_peered":"2023-08-22T23:44:38.880231+0000","last_clean":"2023-08-22T23:44:38.880231+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:44:38.880231+0000","last_undegraded":"2023-08-22T23:44:38.880231+0000","last_fullsized":"2023-08-22T23:44:38.880231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:29:31.117622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":866,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.293455+0000","last_change":"2023-08-22T23:23:40.333373+0000","last_active":"2023-08-22T23:44:34.293455+0000","last_peered":"2023-08-22T23:44:34.293455+0000","last_clean":"2023-08-22T23:44:34.293455+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:34.293455+0000","last_undegraded":"2023-08-22T23:44:34.293455+0000","last_fullsized":"2023-08-22T23:44:34.293455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:44:22.179832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":875,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.602654+0000","last_change":"2023-08-22T23:23:35.301428+0000","last_active":"2023-08-22T23:44:36.602654+0000","last_peered":"2023-08-22T23:44:36.602654+0000","last_clean":"2023-08-22T23:44:36.602654+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:44:36.602654+0000","last_undegraded":"2023-08-22T23:44:36.602654+0000","last_fullsized":"2023-08-22T23:44:36.602654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:04:58.620425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":866,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.296412+0000","last_change":"2023-08-22T23:23:42.559066+0000","last_active":"2023-08-22T23:44:34.296412+0000","last_peered":"2023-08-22T23:44:34.296412+0000","last_clean":"2023-08-22T23:44:34.296412+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:34.296412+0000","last_undegraded":"2023-08-22T23:44:34.296412+0000","last_fullsized":"2023-08-22T23:44:34.296412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:59:53.939044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":874,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.556608+0000","last_change":"2023-08-22T23:23:39.799381+0000","last_active":"2023-08-22T23:44:36.556608+0000","last_peered":"2023-08-22T23:44:36.556608+0000","last_clean":"2023-08-22T23:44:36.556608+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:36.556608+0000","last_undegraded":"2023-08-22T23:44:36.556608+0000","last_fullsized":"2023-08-22T23:44:36.556608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:45:47.412302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":863,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.335096+0000","last_change":"2023-08-22T23:23:31.473444+0000","last_active":"2023-08-22T23:44:34.335096+0000","last_peered":"2023-08-22T23:44:34.335096+0000","last_clean":"2023-08-22T23:44:34.335096+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:34.335096+0000","last_undegraded":"2023-08-22T23:44:34.335096+0000","last_fullsized":"2023-08-22T23:44:34.335096+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:41:55.891550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":861,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.446796+0000","last_change":"2023-08-22T23:23:30.881230+0000","last_active":"2023-08-22T23:44:34.446796+0000","last_peered":"2023-08-22T23:44:34.446796+0000","last_clean":"2023-08-22T23:44:34.446796+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:44:34.446796+0000","last_undegraded":"2023-08-22T23:44:34.446796+0000","last_fullsized":"2023-08-22T23:44:34.446796+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:45:22.079165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":877,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.282867+0000","last_change":"2023-08-22T23:23:57.358686+0000","last_active":"2023-08-22T23:44:34.282867+0000","last_peered":"2023-08-22T23:44:34.282867+0000","last_clean":"2023-08-22T23:44:34.282867+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:34.282867+0000","last_undegraded":"2023-08-22T23:44:34.282867+0000","last_fullsized":"2023-08-22T23:44:34.282867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:14:12.541623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":863,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.228073+0000","last_change":"2023-08-22T23:23:36.215630+0000","last_active":"2023-08-22T23:44:37.228073+0000","last_peered":"2023-08-22T23:44:37.228073+0000","last_clean":"2023-08-22T23:44:37.228073+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:37.228073+0000","last_undegraded":"2023-08-22T23:44:37.228073+0000","last_fullsized":"2023-08-22T23:44:37.228073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:45:18.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.859661+0000","last_change":"2023-08-22T23:23:45.579934+0000","last_active":"2023-08-22T23:44:39.859661+0000","last_peered":"2023-08-22T23:44:39.859661+0000","last_clean":"2023-08-22T23:44:39.859661+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:44:39.859661+0000","last_undegraded":"2023-08-22T23:44:39.859661+0000","last_fullsized":"2023-08-22T23:44:39.859661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:22:30.109602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.896557+0000","last_change":"2023-08-22T23:22:30.241386+0000","last_active":"2023-08-22T23:44:38.896557+0000","last_peered":"2023-08-22T23:44:38.896557+0000","last_clean":"2023-08-22T23:44:38.896557+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:44:38.896557+0000","last_undegraded":"2023-08-22T23:44:38.896557+0000","last_fullsized":"2023-08-22T23:44:38.896557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:44:37.084613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":873,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.199160+0000","last_change":"2023-08-22T23:23:37.170814+0000","last_active":"2023-08-22T23:44:34.199160+0000","last_peered":"2023-08-22T23:44:34.199160+0000","last_clean":"2023-08-22T23:44:34.199160+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:34.199160+0000","last_undegraded":"2023-08-22T23:44:34.199160+0000","last_fullsized":"2023-08-22T23:44:34.199160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:09:15.576464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":878,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.189811+0000","last_change":"2023-08-22T23:22:30.276299+0000","last_active":"2023-08-22T23:44:34.189811+0000","last_peered":"2023-08-22T23:44:34.189811+0000","last_clean":"2023-08-22T23:44:34.189811+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:34.189811+0000","last_undegraded":"2023-08-22T23:44:34.189811+0000","last_fullsized":"2023-08-22T23:44:34.189811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-23T23:36:37.347305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":874,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.205810+0000","last_change":"2023-08-22T23:23:32.260717+0000","last_active":"2023-08-22T23:44:34.205810+0000","last_peered":"2023-08-22T23:44:34.205810+0000","last_clean":"2023-08-22T23:44:34.205810+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:34.205810+0000","last_undegraded":"2023-08-22T23:44:34.205810+0000","last_fullsized":"2023-08-22T23:44:34.205810+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:34:26.225961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.886114+0000","last_change":"2023-08-22T23:22:30.036499+0000","last_active":"2023-08-22T23:44:38.886114+0000","last_peered":"2023-08-22T23:44:38.886114+0000","last_clean":"2023-08-22T23:44:38.886114+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:44:38.886114+0000","last_undegraded":"2023-08-22T23:44:38.886114+0000","last_fullsized":"2023-08-22T23:44:38.886114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:38:38.054037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":857,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.259759+0000","last_change":"2023-08-22T23:23:47.632982+0000","last_active":"2023-08-22T23:44:34.259759+0000","last_peered":"2023-08-22T23:44:34.259759+0000","last_clean":"2023-08-22T23:44:34.259759+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:34.259759+0000","last_undegraded":"2023-08-22T23:44:34.259759+0000","last_fullsized":"2023-08-22T23:44:34.259759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:55:33.947607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":867,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.366073+0000","last_change":"2023-08-22T23:22:30.442903+0000","last_active":"2023-08-22T23:44:34.366073+0000","last_peered":"2023-08-22T23:44:34.366073+0000","last_clean":"2023-08-22T23:44:34.366073+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:34.366073+0000","last_undegraded":"2023-08-22T23:44:34.366073+0000","last_fullsized":"2023-08-22T23:44:34.366073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:53:41.101184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":879,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.315380+0000","last_change":"2023-08-22T23:23:44.752571+0000","last_active":"2023-08-22T23:44:34.315380+0000","last_peered":"2023-08-22T23:44:34.315380+0000","last_clean":"2023-08-22T23:44:34.315380+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:34.315380+0000","last_undegraded":"2023-08-22T23:44:34.315380+0000","last_fullsized":"2023-08-22T23:44:34.315380+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:11:41.512409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":865,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.161011+0000","last_change":"2023-08-22T23:23:43.800737+0000","last_active":"2023-08-22T23:44:37.161011+0000","last_peered":"2023-08-22T23:44:37.161011+0000","last_clean":"2023-08-22T23:44:37.161011+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:37.161011+0000","last_undegraded":"2023-08-22T23:44:37.161011+0000","last_fullsized":"2023-08-22T23:44:37.161011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:51:40.546608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.858703+0000","last_change":"2023-08-22T23:22:30.393248+0000","last_active":"2023-08-22T23:44:38.858703+0000","last_peered":"2023-08-22T23:44:38.858703+0000","last_clean":"2023-08-22T23:44:38.858703+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:38.858703+0000","last_undegraded":"2023-08-22T23:44:38.858703+0000","last_fullsized":"2023-08-22T23:44:38.858703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:16:05.115250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":865,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.332085+0000","last_change":"2023-08-22T23:23:31.308224+0000","last_active":"2023-08-22T23:44:34.332085+0000","last_peered":"2023-08-22T23:44:34.332085+0000","last_clean":"2023-08-22T23:44:34.332085+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:34.332085+0000","last_undegraded":"2023-08-22T23:44:34.332085+0000","last_fullsized":"2023-08-22T23:44:34.332085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:16:42.335154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":877,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.147945+0000","last_change":"2023-08-22T23:22:31.576502+0000","last_active":"2023-08-22T23:44:34.147945+0000","last_peered":"2023-08-22T23:44:34.147945+0000","last_clean":"2023-08-22T23:44:34.147945+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:34.147945+0000","last_undegraded":"2023-08-22T23:44:34.147945+0000","last_fullsized":"2023-08-22T23:44:34.147945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:26:48.005613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"101'7","reported_seq":870,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.311560+0000","last_change":"2023-08-22T23:23:56.020996+0000","last_active":"2023-08-22T23:44:34.311560+0000","last_peered":"2023-08-22T23:44:34.311560+0000","last_clean":"2023-08-22T23:44:34.311560+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:34.311560+0000","last_undegraded":"2023-08-22T23:44:34.311560+0000","last_fullsized":"2023-08-22T23:44:34.311560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:06:07.542557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.842303+0000","last_change":"2023-08-22T23:22:29.967457+0000","last_active":"2023-08-22T23:44:38.842303+0000","last_peered":"2023-08-22T23:44:38.842303+0000","last_clean":"2023-08-22T23:44:38.842303+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:38.842303+0000","last_undegraded":"2023-08-22T23:44:38.842303+0000","last_fullsized":"2023-08-22T23:44:38.842303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:34:10.033572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":877,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.572507+0000","last_change":"2023-08-22T23:23:34.106212+0000","last_active":"2023-08-22T23:44:36.572507+0000","last_peered":"2023-08-22T23:44:36.572507+0000","last_clean":"2023-08-22T23:44:36.572507+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:36.572507+0000","last_undegraded":"2023-08-22T23:44:36.572507+0000","last_fullsized":"2023-08-22T23:44:36.572507+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:18:39.449835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":887,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.870022+0000","last_change":"2023-08-22T23:22:30.329629+0000","last_active":"2023-08-22T23:44:38.870022+0000","last_peered":"2023-08-22T23:44:38.870022+0000","last_clean":"2023-08-22T23:44:38.870022+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:38.870022+0000","last_undegraded":"2023-08-22T23:44:38.870022+0000","last_fullsized":"2023-08-22T23:44:38.870022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:44:32.527595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":979,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.277016+0000","last_change":"2023-08-22T23:22:24.344727+0000","last_active":"2023-08-22T23:44:34.277016+0000","last_peered":"2023-08-22T23:44:34.277016+0000","last_clean":"2023-08-22T23:44:34.277016+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:34.277016+0000","last_undegraded":"2023-08-22T23:44:34.277016+0000","last_fullsized":"2023-08-22T23:44:34.277016+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_clean_scrub_stamp":"2023-08-22T23:22:21.305061+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:48:33.189247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":876,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.521756+0000","last_change":"2023-08-22T23:23:37.130773+0000","last_active":"2023-08-22T23:44:36.521756+0000","last_peered":"2023-08-22T23:44:36.521756+0000","last_clean":"2023-08-22T23:44:36.521756+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:44:36.521756+0000","last_undegraded":"2023-08-22T23:44:36.521756+0000","last_fullsized":"2023-08-22T23:44:36.521756+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:03:26.536335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.847871+0000","last_change":"2023-08-22T23:23:33.761439+0000","last_active":"2023-08-22T23:44:38.847871+0000","last_peered":"2023-08-22T23:44:38.847871+0000","last_clean":"2023-08-22T23:44:38.847871+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:44:38.847871+0000","last_undegraded":"2023-08-22T23:44:38.847871+0000","last_fullsized":"2023-08-22T23:44:38.847871+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:21:42.148901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.855176+0000","last_change":"2023-08-22T23:23:33.024596+0000","last_active":"2023-08-22T23:44:38.855176+0000","last_peered":"2023-08-22T23:44:38.855176+0000","last_clean":"2023-08-22T23:44:38.855176+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:44:38.855176+0000","last_undegraded":"2023-08-22T23:44:38.855176+0000","last_fullsized":"2023-08-22T23:44:38.855176+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:01:54.420757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":872,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.157343+0000","last_change":"2023-08-22T23:23:36.066660+0000","last_active":"2023-08-22T23:44:34.157343+0000","last_peered":"2023-08-22T23:44:34.157343+0000","last_clean":"2023-08-22T23:44:34.157343+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:34.157343+0000","last_undegraded":"2023-08-22T23:44:34.157343+0000","last_fullsized":"2023-08-22T23:44:34.157343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:53:08.491032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.189231+0000","last_change":"2023-08-22T23:23:32.334749+0000","last_active":"2023-08-22T23:44:37.189231+0000","last_peered":"2023-08-22T23:44:37.189231+0000","last_clean":"2023-08-22T23:44:37.189231+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:37.189231+0000","last_undegraded":"2023-08-22T23:44:37.189231+0000","last_fullsized":"2023-08-22T23:44:37.189231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:23:58.961035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.883169+0000","last_change":"2023-08-22T23:23:32.954050+0000","last_active":"2023-08-22T23:44:38.883169+0000","last_peered":"2023-08-22T23:44:38.883169+0000","last_clean":"2023-08-22T23:44:38.883169+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:44:38.883169+0000","last_undegraded":"2023-08-22T23:44:38.883169+0000","last_fullsized":"2023-08-22T23:44:38.883169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:37:01.389521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":878,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.285757+0000","last_change":"2023-08-22T23:23:55.264729+0000","last_active":"2023-08-22T23:44:34.285757+0000","last_peered":"2023-08-22T23:44:34.285757+0000","last_clean":"2023-08-22T23:44:34.285757+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:34.285757+0000","last_undegraded":"2023-08-22T23:44:34.285757+0000","last_fullsized":"2023-08-22T23:44:34.285757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:35:59.339638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.255436+0000","last_change":"2023-08-22T23:23:39.717617+0000","last_active":"2023-08-22T23:44:37.255436+0000","last_peered":"2023-08-22T23:44:37.255436+0000","last_clean":"2023-08-22T23:44:37.255436+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:37.255436+0000","last_undegraded":"2023-08-22T23:44:37.255436+0000","last_fullsized":"2023-08-22T23:44:37.255436+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:01:30.780516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":867,"reported_epoch":913,"state":"active+clean","last_fresh":"2023-08-22T23:44:34.279154+0000","last_change":"2023-08-22T23:23:43.444588+0000","last_active":"2023-08-22T23:44:34.279154+0000","last_peered":"2023-08-22T23:44:34.279154+0000","last_clean":"2023-08-22T23:44:34.279154+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:34.279154+0000","last_undegraded":"2023-08-22T23:44:34.279154+0000","last_fullsized":"2023-08-22T23:44:34.279154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:36:31.111975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":882,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.846100+0000","last_change":"2023-08-22T23:23:45.510476+0000","last_active":"2023-08-22T23:44:38.846100+0000","last_peered":"2023-08-22T23:44:38.846100+0000","last_clean":"2023-08-22T23:44:38.846100+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:44:38.846100+0000","last_undegraded":"2023-08-22T23:44:38.846100+0000","last_fullsized":"2023-08-22T23:44:38.846100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:30:55.850897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":856,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-08-22T23:44:36.292536+0000","last_change":"2023-08-22T23:23:33.850083+0000","last_active":"2023-08-22T23:44:36.292536+0000","last_peered":"2023-08-22T23:44:36.292536+0000","last_clean":"2023-08-22T23:44:36.292536+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:36.292536+0000","last_undegraded":"2023-08-22T23:44:36.292536+0000","last_fullsized":"2023-08-22T23:44:36.292536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:48:04.458489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.847155+0000","last_change":"2023-08-22T23:23:41.098969+0000","last_active":"2023-08-22T23:44:39.847155+0000","last_peered":"2023-08-22T23:44:39.847155+0000","last_clean":"2023-08-22T23:44:39.847155+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:44:39.847155+0000","last_undegraded":"2023-08-22T23:44:39.847155+0000","last_fullsized":"2023-08-22T23:44:39.847155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:52:09.641904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477004,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66904,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94304936,"statfs":{"total":96636764160,"available":96568254464,"internally_reserved":0,"allocated":13975552,"data_stored":8701029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":13.682,"5min":16.966000000000001,"15min":19.555},"min":{"1min":7.7030000000000003,"5min":4.734,"15min":4.734},"max":{"1min":40.639000000000003,"5min":187.08799999999999,"15min":254.63},"last":7.3460000000000001},{"interface":"front","average":{"1min":17.207999999999998,"5min":23.309999999999999,"15min":22.024999999999999},"min":{"1min":6.1360000000000001,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":47.487000000000002,"5min":330.00099999999998,"15min":330.00099999999998},"last":30.376000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":22.524999999999999,"15min":22.077000000000002},"min":{"1min":6.8159999999999998,"5min":6.8099999999999996,"15min":6.3019999999999996},"max":{"1min":52.658000000000001,"5min":176.803,"15min":176.803},"last":10.028},{"interface":"front","average":{"1min":33.201000000000001,"5min":28.675999999999998,"15min":25.103000000000002},"min":{"1min":8.7319999999999993,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":267.399,"5min":277.048,"15min":277.048},"last":24.405000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.978000000000002,"5min":15.691000000000001,"15min":19.866},"min":{"1min":7.7519999999999998,"5min":6.3860000000000001,"15min":5.766},"max":{"1min":112.148,"5min":112.148,"15min":332.09199999999998},"last":11.475},{"interface":"front","average":{"1min":18.84,"5min":24.292000000000002,"15min":25.629000000000001},"min":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":75.138000000000005,"5min":193.38499999999999,"15min":218.94900000000001},"last":25.276}]},{"osd":4,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":21.919,"5min":22.414999999999999,"15min":25.763999999999999},"min":{"1min":9.3780000000000001,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":107.697,"5min":107.697,"15min":333.517},"last":12.913},{"interface":"front","average":{"1min":21.629000000000001,"5min":28.734000000000002,"15min":29.236000000000001},"min":{"1min":9.7230000000000008,"5min":9.1370000000000005,"15min":8.5779999999999994},"max":{"1min":108.776,"5min":273.65800000000002,"15min":342.31999999999999},"last":29.466999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":22.91,"5min":21.954999999999998,"15min":23.404},"min":{"1min":11.204000000000001,"5min":6.0869999999999997,"15min":6.0869999999999997},"max":{"1min":73.780000000000001,"5min":106.52,"15min":326.71300000000002},"last":26.649000000000001},{"interface":"front","average":{"1min":20.088999999999999,"5min":24.102,"15min":27.399999999999999},"min":{"1min":7.4470000000000001,"5min":7.4470000000000001,"15min":7.2830000000000004},"max":{"1min":99.808000000000007,"5min":196.352,"15min":250.43600000000001},"last":10.839}]},{"osd":6,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.93,"5min":23.151,"15min":26.940999999999999},"min":{"1min":10.054,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":50.091999999999999,"5min":110.843,"15min":267.84800000000001},"last":12.263},{"interface":"front","average":{"1min":24.856000000000002,"5min":30.155999999999999,"15min":29.498999999999999},"min":{"1min":12.632,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":103.387,"5min":198.358,"15min":255.58699999999999},"last":31.390000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":23.925999999999998,"5min":33.804000000000002,"15min":32.649000000000001},"min":{"1min":13.246,"5min":11.016,"15min":10.459},"max":{"1min":76.736999999999995,"5min":274.60899999999998,"15min":274.60899999999998},"last":32.738999999999997},{"interface":"front","average":{"1min":22.082000000000001,"5min":22.986999999999998,"15min":24.119},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":7.2030000000000003},"max":{"1min":98.132999999999996,"5min":114.447,"15min":213.56899999999999},"last":8.4269999999999996}]}]},{"osd":3,"up_from":16,"seq":68719477006,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84584,"kb_used_data":12884,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287256,"statfs":{"total":96636764160,"available":96550150144,"internally_reserved":0,"allocated":13193216,"data_stored":8065623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":17.213000000000001,"5min":15.962999999999999,"15min":21.890000000000001},"min":{"1min":6.5060000000000002,"5min":5.2729999999999997,"15min":5.0359999999999996},"max":{"1min":122.562,"5min":153.98599999999999,"15min":274.45100000000002},"last":11.135},{"interface":"front","average":{"1min":27.960000000000001,"5min":23.204999999999998,"15min":29.393999999999998},"min":{"1min":9.2579999999999991,"5min":7.4489999999999998,"15min":5.2670000000000003},"max":{"1min":215.34800000000001,"5min":215.34800000000001,"15min":356.75599999999997},"last":12.693}]},{"osd":1,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":13.138999999999999,"5min":16.411999999999999,"15min":25.859999999999999},"min":{"1min":5.218,"5min":5.141,"15min":5.141},"max":{"1min":56.652999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":11.866},{"interface":"front","average":{"1min":14.827999999999999,"5min":16.405000000000001,"15min":24.341000000000001},"min":{"1min":6.8529999999999998,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":51.414999999999999,"5min":190.49799999999999,"15min":374.01799999999997},"last":15.992000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":23.094000000000001,"5min":21.440000000000001,"15min":27.952000000000002},"min":{"1min":6.6280000000000001,"5min":6.524,"15min":6.524},"max":{"1min":145.535,"5min":198.649,"15min":353.66000000000003},"last":18.484000000000002},{"interface":"front","average":{"1min":20.449999999999999,"5min":21.263999999999999,"15min":30.486999999999998},"min":{"1min":7.3970000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":108.08,"5min":193.53999999999999,"15min":385.17599999999999},"last":14.114000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.568000000000001,"5min":22.332999999999998,"15min":30.498999999999999},"min":{"1min":9.4250000000000007,"5min":7.2000000000000002,"15min":6.6230000000000002},"max":{"1min":86.454999999999998,"5min":199.42400000000001,"15min":370.84399999999999},"last":21.204999999999998},{"interface":"front","average":{"1min":19.699000000000002,"5min":25.030000000000001,"15min":35.835000000000001},"min":{"1min":8.6219999999999999,"5min":6.726,"15min":6.726},"max":{"1min":64.091999999999999,"5min":231.53,"15min":372.45299999999997},"last":17.559999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":22.189,"15min":32.162999999999997},"min":{"1min":9.7609999999999992,"5min":9.5820000000000007,"15min":8.5419999999999998},"max":{"1min":52.338999999999999,"5min":175.429,"15min":369.33499999999998},"last":25.120999999999999},{"interface":"front","average":{"1min":18.379999999999999,"5min":21.294,"15min":29.463000000000001},"min":{"1min":7.9320000000000004,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":54.829000000000001,"5min":212.09,"15min":289.34300000000002},"last":19.861999999999998}]},{"osd":6,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":18.791,"5min":22.777999999999999,"15min":33.335999999999999},"min":{"1min":8.8979999999999997,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":57.430999999999997,"5min":213.15000000000001,"15min":372.63799999999998},"last":15.125},{"interface":"front","average":{"1min":24.143999999999998,"5min":27.623999999999999,"15min":31.969000000000001},"min":{"1min":14.032,"5min":8.1010000000000009,"15min":7.7830000000000004},"max":{"1min":63.143000000000001,"5min":215.41300000000001,"15min":346.69900000000001},"last":20.538}]},{"osd":7,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.388000000000002,"5min":23.713999999999999,"15min":33.411000000000001},"min":{"1min":10.506,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":58.783999999999999,"5min":206.708,"15min":417.863},"last":22.695},{"interface":"front","average":{"1min":23.991,"5min":28.187999999999999,"15min":31.818999999999999},"min":{"1min":13.135999999999999,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":61.673000000000002,"5min":229.85900000000001,"15min":241.38200000000001},"last":23.719000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66488,"kb_used_data":13220,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305352,"statfs":{"total":96636764160,"available":96568680448,"internally_reserved":0,"allocated":13537280,"data_stored":8257796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":8.4800000000000004,"5min":16.053999999999998,"15min":17.029},"min":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":15.577999999999999,"5min":204.17500000000001,"15min":208.114},"last":21.135000000000002},{"interface":"front","average":{"1min":16.081,"5min":21.925999999999998,"15min":20.050000000000001},"min":{"1min":8.7599999999999998,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":44.661000000000001,"5min":205.833,"15min":205.833},"last":28.797000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":17.501000000000001,"15min":18.324999999999999},"min":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":114.821,"5min":142.42599999999999,"15min":218.06200000000001},"last":7.5439999999999996},{"interface":"front","average":{"1min":24.239999999999998,"5min":22.905000000000001,"15min":21.925999999999998},"min":{"1min":10.121,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":115.809,"5min":145.83000000000001,"15min":230.108},"last":31.318999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:43:54 2023","interfaces":[{"interface":"back","average":{"1min":16.317,"5min":21.879000000000001,"15min":21.068999999999999},"min":{"1min":7.0880000000000001,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":102.65300000000001,"5min":292.49000000000001,"15min":292.49000000000001},"last":8.4860000000000007},{"interface":"front","average":{"1min":20.469999999999999,"5min":28.943999999999999,"15min":25.609000000000002},"min":{"1min":9.5030000000000001,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":107.33799999999999,"5min":304.017,"15min":306.26299999999998},"last":25.576000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":15.987,"5min":19.297999999999998,"15min":21.773},"min":{"1min":8.7170000000000005,"5min":8.3759999999999994,"15min":7.3869999999999996},"max":{"1min":52.591999999999999,"5min":157.559,"15min":187.357},"last":24.66},{"interface":"front","average":{"1min":14.167999999999999,"5min":18.696999999999999,"15min":20.98},"min":{"1min":7.7050000000000001,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":19.834,"5min":146.64599999999999,"15min":206.59100000000001},"last":21.797999999999998}]},{"osd":5,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.920000000000002,"5min":22.468,"15min":24.614000000000001},"min":{"1min":10.135,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":54.234000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":29.536999999999999},{"interface":"front","average":{"1min":22.023,"5min":21.733000000000001,"15min":23.268000000000001},"min":{"1min":8.9309999999999992,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":63.061999999999998,"5min":181.24199999999999,"15min":181.24199999999999},"last":26.881}]},{"osd":6,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":27.670999999999999,"5min":25.062999999999999,"15min":24.218},"min":{"1min":11.868,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":127.512,"5min":189.55000000000001,"15min":189.55000000000001},"last":30.242000000000001},{"interface":"front","average":{"1min":25.172999999999998,"5min":21.625,"15min":24.170000000000002},"min":{"1min":9.8800000000000008,"5min":9.0280000000000005,"15min":8.8819999999999997},"max":{"1min":119.714,"5min":129.928,"15min":187.434},"last":26.263999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":26.472999999999999,"15min":29.672000000000001},"min":{"1min":11.529999999999999,"5min":10.513,"15min":9.9939999999999998},"max":{"1min":55.665999999999997,"5min":195.655,"15min":402.06099999999998},"last":33.277000000000001},{"interface":"front","average":{"1min":17.021000000000001,"5min":21.558,"15min":28.571999999999999},"min":{"1min":8.9019999999999992,"5min":8.6690000000000005,"15min":8.6690000000000005},"max":{"1min":39.817,"5min":158.26499999999999,"15min":427.274},"last":34.084000000000003}]}]},{"osd":0,"up_from":16,"seq":68719477006,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66528,"kb_used_data":13248,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305312,"statfs":{"total":96636764160,"available":96568639488,"internally_reserved":0,"allocated":13565952,"data_stored":8272645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.164,"15min":17.242999999999999},"min":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":28.050000000000001,"5min":125.212,"15min":315.35300000000001},"last":30.100999999999999},{"interface":"front","average":{"1min":11.481999999999999,"5min":19.741,"15min":19.794},"min":{"1min":5.8739999999999997,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":27.388000000000002,"5min":150.68199999999999,"15min":314.375},"last":28.806000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":15.347,"5min":22.693000000000001,"15min":21.102},"min":{"1min":7.5519999999999996,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":42.912999999999997,"5min":133.678,"15min":255.041},"last":10.315},{"interface":"front","average":{"1min":14.930999999999999,"5min":23.047000000000001,"15min":21.428000000000001},"min":{"1min":7.1859999999999999,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":31.559999999999999,"5min":124.077,"15min":266.08600000000001},"last":12.804}]},{"osd":3,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":14.122,"5min":21.184999999999999,"15min":20.628},"min":{"1min":7.6429999999999998,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":77.963999999999999,"5min":140.357,"15min":359.37},"last":11.356999999999999},{"interface":"front","average":{"1min":20.382000000000001,"5min":23.077999999999999,"15min":22.991},"min":{"1min":8.3200000000000003,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":80.066999999999993,"5min":133.27000000000001,"15min":335.35399999999998},"last":12.057}]},{"osd":4,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":17.265999999999998,"5min":21.844999999999999,"15min":22.927},"min":{"1min":9.0359999999999996,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":40.703000000000003,"5min":270.06900000000002,"15min":270.06900000000002},"last":13.423},{"interface":"front","average":{"1min":17.626999999999999,"5min":23.300000000000001,"15min":23.582000000000001},"min":{"1min":7.2880000000000003,"5min":7.1689999999999996,"15min":7.1689999999999996},"max":{"1min":48.825000000000003,"5min":204.54499999999999,"15min":301.78399999999999},"last":15.983000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":18.289999999999999,"5min":24.518000000000001,"15min":25.202999999999999},"min":{"1min":11.141,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":30.463000000000001,"5min":153.78100000000001,"15min":287.80200000000002},"last":263.81599999999997},{"interface":"front","average":{"1min":19.331,"5min":24.251000000000001,"15min":26.376000000000001},"min":{"1min":9.8699999999999992,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":37.241,"5min":130.245,"15min":260.29399999999998},"last":272.935}]},{"osd":6,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":25.445,"5min":28.920999999999999,"15min":25.911000000000001},"min":{"1min":10.363,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":159.46299999999999,"5min":243.15100000000001,"15min":276.935},"last":28.029},{"interface":"front","average":{"1min":21.541,"5min":23.311,"15min":24.834},"min":{"1min":10.583,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":109.886,"5min":241.351,"15min":265.17399999999998},"last":26.350999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":33.301000000000002,"5min":29.783000000000001,"15min":31.905999999999999},"min":{"1min":13.009,"5min":10.702,"15min":7.1760000000000002},"max":{"1min":184.333,"5min":227.852,"15min":770.02200000000005},"last":16.956},{"interface":"front","average":{"1min":31.655000000000001,"5min":27.768000000000001,"15min":31.879999999999999},"min":{"1min":10.598000000000001,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":154.00800000000001,"5min":209.393,"15min":834.71799999999996},"last":17.928000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509710,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66532,"kb_used_data":13264,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305308,"statfs":{"total":96636764160,"available":96568635392,"internally_reserved":0,"allocated":13582336,"data_stored":8302852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":20.427,"5min":17.768000000000001,"15min":18.873999999999999},"min":{"1min":7.3470000000000004,"5min":6.2119999999999997,"15min":4.5570000000000004},"max":{"1min":95.617000000000004,"5min":130.898,"15min":358.06400000000002},"last":6.9870000000000001},{"interface":"front","average":{"1min":24.739999999999998,"5min":24.744,"15min":22.606999999999999},"min":{"1min":10.959,"5min":9.2430000000000003,"15min":4.9699999999999998},"max":{"1min":103.443,"5min":146.55500000000001,"15min":233.333},"last":9.0399999999999991}]},{"osd":1,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.175999999999998,"5min":15.595000000000001,"15min":17.597000000000001},"min":{"1min":6.4939999999999998,"5min":5.1100000000000003,"15min":5.1100000000000003},"max":{"1min":63.090000000000003,"5min":112.752,"15min":222.346},"last":22.611000000000001},{"interface":"front","average":{"1min":20.001999999999999,"5min":19.123000000000001,"15min":20.712},"min":{"1min":8.8339999999999996,"5min":8.0579999999999998,"15min":6.0519999999999996},"max":{"1min":87.179000000000002,"5min":137.86799999999999,"15min":258.90600000000001},"last":8.3179999999999996}]},{"osd":2,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":21.442,"5min":18.684000000000001,"15min":19.425000000000001},"min":{"1min":6.6619999999999999,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":92.921000000000006,"5min":171.38399999999999,"15min":283.65600000000001},"last":28.452999999999999},{"interface":"front","average":{"1min":22.297999999999998,"5min":21.972000000000001,"15min":23.984999999999999},"min":{"1min":9.0169999999999995,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":100.437,"5min":170.02600000000001,"15min":389.19299999999998},"last":33.014000000000003}]},{"osd":3,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":22.475999999999999,"15min":27.465},"min":{"1min":7.6630000000000003,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":208.94800000000001,"5min":214.666,"15min":352.79500000000002},"last":36.924999999999997},{"interface":"front","average":{"1min":25.527000000000001,"5min":23.891999999999999,"15min":24.518000000000001},"min":{"1min":10.247999999999999,"5min":7.4370000000000003,"15min":7.415},"max":{"1min":91.753,"5min":178.21600000000001,"15min":314.69},"last":27.300000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.849,"5min":20.013000000000002,"15min":23.686},"min":{"1min":7.3520000000000003,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":91.307000000000002,"5min":121.392,"15min":316.47500000000002},"last":26.631},{"interface":"front","average":{"1min":23.972999999999999,"5min":27.207999999999998,"15min":30.826000000000001},"min":{"1min":12.016999999999999,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":92.557000000000002,"5min":156.40100000000001,"15min":326.65100000000001},"last":36.167999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":24.760999999999999,"15min":26.306000000000001},"min":{"1min":10.253,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":86.361999999999995,"5min":183.268,"15min":249.37299999999999},"last":31.268999999999998},{"interface":"front","average":{"1min":24.760000000000002,"5min":22.363,"15min":23.788},"min":{"1min":9.9090000000000007,"5min":9.2989999999999995,"15min":7.3849999999999998},"max":{"1min":93.683000000000007,"5min":146.96199999999999,"15min":215.76499999999999},"last":34.747999999999998}]},{"osd":7,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.631,"15min":26.940000000000001},"min":{"1min":10.702,"5min":10.388999999999999,"15min":9.3559999999999999},"max":{"1min":46.847999999999999,"5min":135.65700000000001,"15min":457.08199999999999},"last":40.125},{"interface":"front","average":{"1min":19.600999999999999,"5min":26.585999999999999,"15min":26.331},"min":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":54.695,"5min":247.423,"15min":353.75900000000001},"last":32.137}]}]},{"osd":4,"up_from":15,"seq":64424509709,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88676,"kb_used_data":13264,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283164,"statfs":{"total":96636764160,"available":96545959936,"internally_reserved":0,"allocated":13582336,"data_stored":8298077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":20.602,"5min":22.466999999999999,"15min":22.718},"min":{"1min":4.7880000000000003,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":90.212000000000003,"5min":141.732,"15min":235.06100000000001},"last":10.895},{"interface":"front","average":{"1min":25.978000000000002,"5min":24.094000000000001,"15min":27.666},"min":{"1min":7.5999999999999996,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":112.598,"5min":189.517,"15min":395.15300000000002},"last":8.1780000000000008}]},{"osd":1,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":36.859999999999999,"5min":25.800999999999998,"15min":26.106999999999999},"min":{"1min":6.9509999999999996,"5min":6.7149999999999999,"15min":5.7439999999999998},"max":{"1min":228.94499999999999,"5min":241.87799999999999,"15min":382.04199999999997},"last":23.074999999999999},{"interface":"front","average":{"1min":41.487000000000002,"5min":33.701999999999998,"15min":33.277999999999999},"min":{"1min":11.436999999999999,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":237.38499999999999,"5min":252.441,"15min":382.69},"last":26.588999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":23.855,"5min":24.207999999999998,"15min":25.460999999999999},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":214.36500000000001,"5min":219.37899999999999,"15min":537.76199999999994},"last":47.415999999999997},{"interface":"front","average":{"1min":20.466999999999999,"5min":26.917999999999999,"15min":32.433999999999997},"min":{"1min":6.5999999999999996,"5min":6.5999999999999996,"15min":6.1210000000000004},"max":{"1min":140.50200000000001,"5min":214.50700000000001,"15min":556.82899999999995},"last":37.369}]},{"osd":3,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":24.536999999999999,"5min":24.959,"15min":28.465},"min":{"1min":8.3970000000000002,"5min":7.0449999999999999,"15min":6.6159999999999997},"max":{"1min":160.98400000000001,"5min":161.05199999999999,"15min":354.50400000000002},"last":13.960000000000001},{"interface":"front","average":{"1min":31.042999999999999,"5min":30.719999999999999,"15min":30.981999999999999},"min":{"1min":10.749000000000001,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":226.25700000000001,"5min":226.25700000000001,"15min":339.51499999999999},"last":14.798}]},{"osd":5,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":34.338000000000001,"5min":30.631,"15min":32.762999999999998},"min":{"1min":9.1660000000000004,"5min":7.4160000000000004,"15min":7.2279999999999998},"max":{"1min":139.31899999999999,"5min":238.57300000000001,"15min":380.863},"last":155.17599999999999},{"interface":"front","average":{"1min":26.352,"5min":30.120999999999999,"15min":36.542000000000002},"min":{"1min":7.5999999999999996,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":99.632999999999996,"5min":215.928,"15min":364.09500000000003},"last":158.03899999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":32.222000000000001,"5min":30.013999999999999,"15min":31.196000000000002},"min":{"1min":12.083,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":190.446,"5min":217,"15min":249.29599999999999},"last":70.799000000000007},{"interface":"front","average":{"1min":24.103000000000002,"5min":24.334,"15min":31.428000000000001},"min":{"1min":8.952,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":132.864,"5min":220.87100000000001,"15min":468.887},"last":74.850999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.209,"5min":24.561,"15min":30.809000000000001},"min":{"1min":10.443,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":26.222999999999999,"5min":187.30099999999999,"15min":361.45499999999998},"last":11.305999999999999},{"interface":"front","average":{"1min":19.317,"5min":28.436,"15min":31.718},"min":{"1min":14.016999999999999,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":33.298999999999999,"5min":197.05600000000001,"15min":368.08699999999999},"last":12.84}]}]},{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88680,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283160,"statfs":{"total":96636764160,"available":96545955840,"internally_reserved":0,"allocated":13598720,"data_stored":8314461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":11.916,"5min":19.649000000000001,"15min":20.076000000000001},"min":{"1min":5.6710000000000003,"5min":5.6710000000000003,"15min":4.3040000000000003},"max":{"1min":38.225999999999999,"5min":307.584,"15min":307.584},"last":10.816000000000001},{"interface":"front","average":{"1min":16.166,"5min":21.260999999999999,"15min":20.367000000000001},"min":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":48.941000000000003,"5min":317.81,"15min":317.81},"last":13.271000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":13.314,"5min":17.096,"15min":25.003},"min":{"1min":6.2629999999999999,"5min":5.077,"15min":5.077},"max":{"1min":22.173999999999999,"5min":159.58799999999999,"15min":319.34500000000003},"last":18.667000000000002},{"interface":"front","average":{"1min":11.484999999999999,"5min":14.808,"15min":20.055},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":26.338000000000001,"5min":160.74600000000001,"15min":195.78999999999999},"last":6.96}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.140000000000001,"5min":17.152999999999999,"15min":23.173999999999999},"min":{"1min":8.6620000000000008,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":47.432000000000002,"5min":148.97800000000001,"15min":772.00400000000002},"last":11.766},{"interface":"front","average":{"1min":17.105,"5min":20.027999999999999,"15min":24.41},"min":{"1min":10.882,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":43.933999999999997,"5min":151.46700000000001,"15min":789.553},"last":8.6929999999999996}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.602,"5min":19.146000000000001,"15min":22.295999999999999},"min":{"1min":7.7370000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":70.665999999999997,"5min":154.44200000000001,"15min":243.33699999999999},"last":9.9440000000000008},{"interface":"front","average":{"1min":18.843,"5min":24.088999999999999,"15min":26.478000000000002},"min":{"1min":6.7880000000000003,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":68.334000000000003,"5min":337.09699999999998,"15min":337.09699999999998},"last":12.621}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":20.622,"5min":21.59,"15min":29.172999999999998},"min":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.0570000000000004},"max":{"1min":109.88800000000001,"5min":111.015,"15min":333.04000000000002},"last":17.716999999999999},{"interface":"front","average":{"1min":18.895,"5min":20.015999999999998,"15min":24.294},"min":{"1min":8.7140000000000004,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":97.224999999999994,"5min":117.06100000000001,"15min":240.084},"last":13.933}]},{"osd":5,"last update":"Tue Aug 22 23:44:05 2023","interfaces":[{"interface":"back","average":{"1min":20.968,"5min":24.206,"15min":30.463999999999999},"min":{"1min":9.9779999999999998,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":60.380000000000003,"5min":247.898,"15min":342.53100000000001},"last":14.548},{"interface":"front","average":{"1min":19.407,"5min":24.077999999999999,"15min":27.297999999999998},"min":{"1min":8.8770000000000007,"5min":8.8770000000000007,"15min":7.4749999999999996},"max":{"1min":49.686,"5min":176.309,"15min":271.47500000000002},"last":20.128}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.318000000000001,"5min":21.977,"15min":25.302},"min":{"1min":8.5459999999999994,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":32.176000000000002,"5min":144.21600000000001,"15min":228.43100000000001},"last":19.504000000000001},{"interface":"front","average":{"1min":17.408000000000001,"5min":23.198,"15min":28.206},"min":{"1min":10.394,"5min":10.090999999999999,"15min":9.2650000000000006},"max":{"1min":24.224,"5min":141.92599999999999,"15min":262.45800000000003},"last":16.774999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542412,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85044,"kb_used_data":13716,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94286796,"statfs":{"total":96636764160,"available":96549679104,"internally_reserved":0,"allocated":14045184,"data_stored":8766223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":87,"apply_latency_ms":87,"commit_latency_ns":87000000,"apply_latency_ns":87000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":12.601000000000001,"5min":16.588000000000001,"15min":20.968},"min":{"1min":6.7779999999999996,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":33.573999999999998,"5min":114.51300000000001,"15min":220.017},"last":9.0960000000000001},{"interface":"front","average":{"1min":18.562000000000001,"5min":22.257999999999999,"15min":27.212},"min":{"1min":10.353999999999999,"5min":6.7919999999999998,"15min":5.9329999999999998},"max":{"1min":32.883000000000003,"5min":115.74299999999999,"15min":264.11900000000003},"last":10.179}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":10.41,"5min":14.409000000000001,"15min":20.576000000000001},"min":{"1min":6.5119999999999996,"5min":6.1289999999999996,"15min":5.9379999999999997},"max":{"1min":16.731000000000002,"5min":102.66200000000001,"15min":343.899},"last":26.073},{"interface":"front","average":{"1min":15.853,"5min":21.071999999999999,"15min":25.216000000000001},"min":{"1min":8.7690000000000001,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":27.713000000000001,"5min":111.873,"15min":270.40499999999997},"last":10.869999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":14.057,"5min":17.219000000000001,"15min":21.003},"min":{"1min":6.4080000000000004,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":28.347000000000001,"5min":116.45,"15min":248.74799999999999},"last":8.125},{"interface":"front","average":{"1min":14.875,"5min":17.582000000000001,"15min":24.184999999999999},"min":{"1min":7.0060000000000002,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":26.632000000000001,"5min":114.843,"15min":256.94400000000002},"last":24.896000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":24.103999999999999,"15min":30.831},"min":{"1min":7.6070000000000002,"5min":6.5739999999999998,"15min":6.5739999999999998},"max":{"1min":50.060000000000002,"5min":231.99799999999999,"15min":257.23500000000001},"last":28.047000000000001},{"interface":"front","average":{"1min":25.657,"5min":23.934999999999999,"15min":24.399000000000001},"min":{"1min":9.8130000000000006,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":62.527999999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":35.063000000000002}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.437999999999999,"5min":25.553000000000001,"15min":28.288},"min":{"1min":9.5950000000000006,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":33.143000000000001,"5min":208.03200000000001,"15min":374.83499999999998},"last":11.807},{"interface":"front","average":{"1min":15.750999999999999,"5min":25.452999999999999,"15min":26.808},"min":{"1min":9.1219999999999999,"5min":7.5389999999999997,"15min":6.3019999999999996},"max":{"1min":22.956,"5min":273.58699999999999,"15min":341.255},"last":30.215}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":21.895,"5min":26.039000000000001,"15min":31.167999999999999},"min":{"1min":9.6560000000000006,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":49.250999999999998,"5min":228.626,"15min":241.26499999999999},"last":35.978000000000002},{"interface":"front","average":{"1min":16.486000000000001,"5min":23.117000000000001,"15min":26.573},"min":{"1min":8.891,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":32.026000000000003,"5min":247.06800000000001,"15min":247.06800000000001},"last":33.091999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":21.434000000000001,"15min":27.062999999999999},"min":{"1min":10.872,"5min":8.2560000000000002,"15min":8.2560000000000002},"max":{"1min":31.568000000000001,"5min":121.524,"15min":317.44600000000003},"last":28.748000000000001},{"interface":"front","average":{"1min":17.907,"5min":25.620000000000001,"15min":28.559999999999999},"min":{"1min":11.589,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":30.829000000000001,"5min":201.81700000000001,"15min":361.93000000000001},"last":31.341000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:44:42.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-22T23:44:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.302+0000 7f198a40d640 10 monclient: get_monmap_and_config 2023-08-22T23:44:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.302+0000 7f198a40d640 10 monclient: build_initial_monmap 2023-08-22T23:44:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.302+0000 7f198a40d640 10 monclient: monmap: 2023-08-22T23:44:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:42.303205+0000 2023-08-22T23:44:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:42.303205+0000 2023-08-22T23:44:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:42.305 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.303+0000 7f198a40d640 1 Processor -- start 2023-08-22T23:44:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.303+0000 7f198a40d640 1 -- start start 2023-08-22T23:44:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.303+0000 7f198a40d640 10 monclient: init 2023-08-22T23:44:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-08-22T23:44:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19840a7b90 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840b1380 0x7f19841417e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient(hunting): picked mon.noname-b con 0x7f19840b1380 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19840a7f70 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19840a9b00 con 0x7f19840a7b90 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19840a9de0 con 0x7f19840a7f70 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19840aa0c0 con 0x7f19840b1380 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f198a40d640 10 monclient(hunting): authenticate will time out at 2846.553768s 2023-08-22T23:44:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f1983fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f19840a7b90 0x7f198413e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48084/0 (socket says 172.21.15.29:48084) 2023-08-22T23:44:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.304+0000 7f1983fff640 1 -- 172.21.15.29:0/3580405074 learned_addr learned my addr 172.21.15.29:0/3580405074 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 1 --2- 172.21.15.29:0/3580405074 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840b1380 0x7f19841417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): get_auth_request con 0x7f19840b1380 auth_method 0 2023-08-22T23:44:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:42.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.305+0000 7f19837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:42.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 10 monclient(hunting): handle_auth_done global_id 16670 payload 274 2023-08-22T23:44:42.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 1 -- 172.21.15.29:0/3580405074 >> v1:172.21.15.52:6789/0 conn(0x7f19840a7f70 legacy=0x7f198414ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 1 -- 172.21.15.29:0/3580405074 >> v1:172.21.15.29:6789/0 conn(0x7f19840a7b90 legacy=0x7f198413e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 1 monclient: found mon.noname-b 2023-08-22T23:44:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.307+0000 7f19837fe640 1 -- 172.21.15.29:0/3580405074 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19841412c0 con 0x7f19840b1380 2023-08-22T23:44:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.309+0000 7f19837fe640 1 --2- 172.21.15.29:0/3580405074 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840b1380 0x7f19841417e0 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f19740097d0 tx=0x7f197402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8a0a7c5744d49c0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 1 -- 172.21.15.29:0/3580405074 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974031890 con 0x7f19840b1380 2023-08-22T23:44:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: dump: 2023-08-22T23:44:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:42.316 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:42.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 1 -- 172.21.15.29:0/3580405074 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19740319f0 con 0x7f19840b1380 2023-08-22T23:44:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f198a40d640 5 monclient: authenticate success, global_id 16670 2023-08-22T23:44:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f198a40d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f198a40d640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 1 -- 172.21.15.29:0/3580405074 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974031d80 con 0x7f19840b1380 2023-08-22T23:44:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f198a40d640 4 set_mon_vals no callback set 2023-08-22T23:44:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.312+0000 7f1982ffd640 10 monclient: dump: 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:42.320 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 10 monclient: shutdown 2023-08-22T23:44:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 -- 172.21.15.29:0/3580405074 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840b1380 msgr2=0x7f19841417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 --2- 172.21.15.29:0/3580405074 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840b1380 0x7f19841417e0 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f19740097d0 tx=0x7f197402b930 comp rx=0 tx=0).stop 2023-08-22T23:44:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 -- 172.21.15.29:0/3580405074 shutdown_connections 2023-08-22T23:44:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 --2- 172.21.15.29:0/3580405074 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840b1380 0x7f19841417e0 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 -- 172.21.15.29:0/3580405074 >> 172.21.15.29:0/3580405074 conn(0x7f19840b6a00 msgr2=0x7f19840b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 -- 172.21.15.29:0/3580405074 shutdown_connections 2023-08-22T23:44:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.313+0000 7f198a40d640 1 -- 172.21.15.29:0/3580405074 wait complete. 2023-08-22T23:44:42.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 10 monclient: build_initial_monmap 2023-08-22T23:44:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 10 monclient: monmap: 2023-08-22T23:44:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 1 librados: starting msgr at 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 1 librados: starting objecter 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 1 Processor -- start 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 1 -- start start 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 1 librados: setting wanted keys 2023-08-22T23:44:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 1 librados: calling monclient init 2023-08-22T23:44:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.314+0000 7f198a40d640 10 monclient: init 2023-08-22T23:44:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient: _add_conns ranks=[2,0,1] 2023-08-22T23:44:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 0x7f1984139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): picked mon.noname-c con 0x7f19840a7b90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19840a7f70 0x7f1984142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): picked mon.noname-a con 0x7f19840a7f70 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19840b1380 0x7f19841d5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): picked mon.noname-b con 0x7f19840b1380 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:42.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19841431e0 con 0x7f19840a7f70 2023-08-22T23:44:42.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:42.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1984142cc0 con 0x7f19840b1380 2023-08-22T23:44:42.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:42.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1984145490 con 0x7f19840a7b90 2023-08-22T23:44:42.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:42.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f198a40d640 10 monclient(hunting): authenticate will time out at 2846.564858s 2023-08-22T23:44:42.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f1983fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 0x7f1984139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.315+0000 7f19837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19840a7f70 0x7f1984142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 0x7f1984139e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34198/0 (socket says 172.21.15.29:34198) 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 1 -- 172.21.15.29:0/3089835197 learned_addr learned my addr 172.21.15.29:0/3089835197 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): get_auth_request con 0x7f19840a7b90 auth_method 0 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:42.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19840b1380 0x7f19841d5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:42.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): get_auth_request con 0x7f19840a7f70 auth_method 0 2023-08-22T23:44:42.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:42.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:42.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:42.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 10 monclient(hunting): get_auth_request con 0x7f19840b1380 auth_method 0 2023-08-22T23:44:42.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:42.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:42.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:42.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:42.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:42.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f1983fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:42.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:42.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:42.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:42.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:42.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.316+0000 7f19889aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:42.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.317+0000 7f19889aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:42.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 10 monclient(hunting): handle_auth_done global_id 16673 payload 824 2023-08-22T23:44:42.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 1 -- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19840b1380 msgr2=0x7f19841d5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19840b1380 0x7f19841d5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 1 -- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19840a7f70 msgr2=0x7f1984142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19840a7f70 0x7f1984142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 1 monclient: found mon.noname-c 2023-08-22T23:44:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.319+0000 7f1983fff640 1 -- 172.21.15.29:0/3089835197 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974009400 con 0x7f19840a7b90 2023-08-22T23:44:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.322+0000 7f1983fff640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 0x7f1984139e60 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f196c00c4a0 tx=0x7f196c00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.322+0000 7f1980ff9640 10 client.?.objecter ms_handle_connect 0x7f19840a7b90 2023-08-22T23:44:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.322+0000 7f1980ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:42.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c005ce0 con 0x7f19840a7b90 2023-08-22T23:44:42.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:42.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:42.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: dump: 2023-08-22T23:44:42.344 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:42.344 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:42.344 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.345 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.345 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:42.345 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:42.345 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:42.346 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:42.346 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:42.346 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:42.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: _finish_auth 0 2023-08-22T23:44:42.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:44:42.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:42.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196c005e40 con 0x7f19840a7b90 2023-08-22T23:44:42.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:42.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c007270 con 0x7f19840a7b90 2023-08-22T23:44:42.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:42.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:42.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f1980ff9640 10 monclient: dump: 2023-08-22T23:44:42.349 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:42.349 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:42.350 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.350 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:42.350 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:42.351 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:42.351 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:42.351 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:42.351 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:42.352 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:42.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 5 monclient: authenticate success, global_id 16673 2023-08-22T23:44:42.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 10 monclient: _renew_subs 2023-08-22T23:44:42.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:42.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f19827fc640 4 set_mon_vals no callback set 2023-08-22T23:44:42.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1984142860 con 0x7f19840a7b90 2023-08-22T23:44:42.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 10 client.16673.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 10 monclient: _renew_subs 2023-08-22T23:44:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19841d6650 con 0x7f19840a7b90 2023-08-22T23:44:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.325+0000 7f198a40d640 1 librados: init done 2023-08-22T23:44:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.326+0000 7f198a40d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.326+0000 7f198a40d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.326+0000 7f198a40d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.326+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198404dbd0 con 0x7f19840a7b90 2023-08-22T23:44:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.346+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f196c007490 con 0x7f19840a7b90 2023-08-22T23:44:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.347+0000 7f1980ff9640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f196406ba30 0x7f196406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.347+0000 7f19837fe640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f196406ba30 0x7f196406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.347+0000 7f19837fe640 10 monclient: get_auth_request con 0x7f196406ba30 auth_method 0 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f19837fe640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f196406ba30 0x7f196406def0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1974036000 tx=0x7f197403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f1980ff9640 10 client.16673.objecter ms_handle_connect 0x7f196406ba30 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 501..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f196c05b0c0 con 0x7f19840a7b90 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f1980ff9640 10 client.16673.objecter ms_dispatch 0x7f1984000b90 osd_map(918..918 src has 501..918) v4 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f1980ff9640 3 client.16673.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f1980ff9640 3 client.16673.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:42.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.348+0000 7f1980ff9640 20 client.16673.objecter dump_active .. 0 homeless 2023-08-22T23:44:42.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.599+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f196c0536f0 con 0x7f19840a7b90 2023-08-22T23:44:42.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.599+0000 7f1980ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:42.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.599+0000 7f1980ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.779+0000 7f198a40d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:44:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.779+0000 7f198a40d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:44:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.779+0000 7f198a40d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.779+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19841379c0 con 0x7f19840a7b90 2023-08-22T23:44:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.816+0000 7f1980ff9640 1 -- 172.21.15.29:0/3089835197 <== mon.2 v2:172.21.15.29:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v918) v1 ==== 74+0+17270 (secure 0 0 0) 0x7f196c017090 con 0x7f19840a7b90 2023-08-22T23:44:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.816+0000 7f1980ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-08-22T23:44:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.816+0000 7f1980ff9640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:44:42.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:44:42.818 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":918,"fsid":"cfd0f96d-1c62-4667-a308-870e3273ade2","created":"2023-08-22T23:21:32.489689+0000","modified":"2023-08-22T23:44:39.725470+0000","last_up_change":"2023-08-22T23:22:22.559235+0000","last_in_change":"2023-08-22T23:21:38.340112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":103,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-22T23:22:20.507839+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-22T23:22:27.474738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-08-22T23:23:27.250039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d051e519-9506-4291-a28a-9400b04f3884","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6809","nonce":1473840652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6811","nonce":1473840652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6816","nonce":1473840652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1473840652},{"type":"v1","addr":"172.21.15.29:6813","nonce":1473840652}]},"public_addr":"172.21.15.29:6809/1473840652","cluster_addr":"172.21.15.29:6811/1473840652","heartbeat_back_addr":"172.21.15.29:6816/1473840652","heartbeat_front_addr":"172.21.15.29:6813/1473840652","state":["exists","up"]},{"osd":1,"uuid":"26caaa51-e454-4283-a40b-44550023da15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6825","nonce":2619884743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6827","nonce":2619884743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6831","nonce":2619884743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2619884743},{"type":"v1","addr":"172.21.15.29:6829","nonce":2619884743}]},"public_addr":"172.21.15.29:6825/2619884743","cluster_addr":"172.21.15.29:6827/2619884743","heartbeat_back_addr":"172.21.15.29:6831/2619884743","heartbeat_front_addr":"172.21.15.29:6829/2619884743","state":["exists","up"]},{"osd":2,"uuid":"0a17913e-6533-4e19-a9c7-edd21971b8e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6801","nonce":1029571786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6803","nonce":1029571786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6807","nonce":1029571786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1029571786},{"type":"v1","addr":"172.21.15.29:6805","nonce":1029571786}]},"public_addr":"172.21.15.29:6801/1029571786","cluster_addr":"172.21.15.29:6803/1029571786","heartbeat_back_addr":"172.21.15.29:6807/1029571786","heartbeat_front_addr":"172.21.15.29:6805/1029571786","state":["exists","up"]},{"osd":3,"uuid":"4f849fcc-3618-4171-b806-1a2e480340f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6817","nonce":4227673673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6819","nonce":4227673673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6823","nonce":4227673673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4227673673},{"type":"v1","addr":"172.21.15.29:6821","nonce":4227673673}]},"public_addr":"172.21.15.29:6817/4227673673","cluster_addr":"172.21.15.29:6819/4227673673","heartbeat_back_addr":"172.21.15.29:6823/4227673673","heartbeat_front_addr":"172.21.15.29:6821/4227673673","state":["exists","up"]},{"osd":4,"uuid":"3039bfc4-a618-40c5-9111-5325782f6646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":907,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6803","nonce":2157117708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6805","nonce":2157117708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6809","nonce":2157117708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":2157117708},{"type":"v1","addr":"172.21.15.52:6807","nonce":2157117708}]},"public_addr":"172.21.15.52:6803/2157117708","cluster_addr":"172.21.15.52:6805/2157117708","heartbeat_back_addr":"172.21.15.52:6809/2157117708","heartbeat_front_addr":"172.21.15.52:6807/2157117708","state":["exists","up"]},{"osd":5,"uuid":"355b1e15-08a3-4837-8b46-fb14f5f272e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6827","nonce":4164832567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6829","nonce":4164832567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6833","nonce":4164832567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":4164832567},{"type":"v1","addr":"172.21.15.52:6831","nonce":4164832567}]},"public_addr":"172.21.15.52:6827/4164832567","cluster_addr":"172.21.15.52:6829/4164832567","heartbeat_back_addr":"172.21.15.52:6833/4164832567","heartbeat_front_addr":"172.21.15.52:6831/4164832567","state":["exists","up"]},{"osd":6,"uuid":"45d94939-76a2-46b3-b31d-f79ee4ffc624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6819","nonce":2556931765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6821","nonce":2556931765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6825","nonce":2556931765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":2556931765},{"type":"v1","addr":"172.21.15.52:6823","nonce":2556931765}]},"public_addr":"172.21.15.52:6819/2556931765","cluster_addr":"172.21.15.52:6821/2556931765","heartbeat_back_addr":"172.21.15.52:6825/2556931765","heartbeat_front_addr":"172.21.15.52:6823/2556931765","state":["exists","up"]},{"osd":7,"uuid":"e79b6fdf-58b5-440a-96b0-da1065376e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":912,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6811","nonce":3454580439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6813","nonce":3454580439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6817","nonce":3454580439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":3454580439},{"type":"v1","addr":"172.21.15.52:6815","nonce":3454580439}]},"public_addr":"172.21.15.52:6811/3454580439","cluster_addr":"172.21.15.52:6813/3454580439","heartbeat_back_addr":"172.21.15.52:6817/3454580439","heartbeat_front_addr":"172.21.15.52:6815/3454580439","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:19.045553+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.775383+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.732376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.515362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.723819+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:18.012554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.629140+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-22T23:22:17.069717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":24,"snaps":[{"begin":9,"length":2}]},{"pool":55,"snaps":[{"begin":11,"length":2}]},{"pool":78,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":90,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":97,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":109,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":175,"snaps":[{"begin":12,"length":1}]},{"pool":184,"snaps":[{"begin":14,"length":1}]},{"pool":190,"snaps":[{"begin":17,"length":1}]},{"pool":192,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-22T23:44:42.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 10 librados: watch_flush enter 2023-08-22T23:44:42.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 10 librados: watch_flush exit 2023-08-22T23:44:42.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 20 client.16673.objecter shutdown clearing up homeless session... 2023-08-22T23:44:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 10 client.16673.objecter successfully canceled tick 2023-08-22T23:44:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f196406ba30 msgr2=0x7f196406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f196406ba30 0x7f196406def0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1974036000 tx=0x7f197403c040 comp rx=0 tx=0).stop 2023-08-22T23:44:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 10 monclient: shutdown 2023-08-22T23:44:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 msgr2=0x7f1984139e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.818+0000 7f198a40d640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 0x7f1984139e60 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f196c00c4a0 tx=0x7f196c00c960 comp rx=0 tx=0).stop 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 shutdown_connections 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f196406ba30 0x7f196406def0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f19840b1380 0x7f19841d5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19840a7f70 0x7f1984142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 --2- 172.21.15.29:0/3089835197 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f19840a7b90 0x7f1984139e60 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 >> 172.21.15.29:0/3089835197 conn(0x7f19840b6a00 msgr2=0x7f1984136af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 shutdown_connections 2023-08-22T23:44:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 -- 172.21.15.29:0/3089835197 wait complete. 2023-08-22T23:44:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:42.819+0000 7f198a40d640 1 librados: shutdown 2023-08-22T23:44:43.490 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:28.774 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.497 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:28.780 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.520 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:28.804 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.557 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:28.854 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.559 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:28.843 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.560 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:28.857 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.562 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:28.846 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.579 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:28.877 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.581 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:28.865 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:43.832 INFO:tasks.ceph:Scrubbing osd.0 2023-08-22T23:44:43.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.943+0000 7f7e02a5e640 10 monclient: get_monmap_and_config 2023-08-22T23:44:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.943+0000 7f7e02a5e640 10 monclient: build_initial_monmap 2023-08-22T23:44:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.943+0000 7f7e02a5e640 10 monclient: monmap: 2023-08-22T23:44:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:43.945101+0000 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:43.945101+0000 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:43.947 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.944+0000 7f7e02a5e640 1 Processor -- start 2023-08-22T23:44:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.944+0000 7f7e02a5e640 1 -- start start 2023-08-22T23:44:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.944+0000 7f7e02a5e640 10 monclient: init 2023-08-22T23:44:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-08-22T23:44:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 0x7f7dfc13f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): picked mon.noname-b con 0x7f7dfc0ad6d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7dfc0ae8c0 0x7f7dfc13bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): picked mon.noname-a con 0x7f7dfc0ae8c0 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7dfc06f5e0 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dfc0a7f60 con 0x7f7dfc06f5e0 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dfc13f000 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc13fa60 con 0x7f7dfc0ad6d0 2023-08-22T23:44:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.945+0000 7f7e02a5e640 10 monclient(hunting): authenticate will time out at 2848.195787s 2023-08-22T23:44:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.946+0000 7f7dfbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 0x7f7dfc13f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.947+0000 7f7dfbfff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 0x7f7dfc13f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34214/0 (socket says 172.21.15.29:34214) 2023-08-22T23:44:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.947+0000 7f7dfbfff640 1 -- 172.21.15.29:0/1899678492 learned_addr learned my addr 172.21.15.29:0/1899678492 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.947+0000 7f7dfbfff640 10 monclient(hunting): get_auth_request con 0x7f7dfc0ad6d0 auth_method 0 2023-08-22T23:44:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.947+0000 7f7dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.947+0000 7f7dfbfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.947+0000 7f7dfbfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.948+0000 7f7dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.948+0000 7f7dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.948+0000 7f7dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.948+0000 7f7dfb7fe640 1 --2- 172.21.15.29:0/1899678492 >> v2:172.21.15.29:3300/0 conn(0x7f7dfc0ae8c0 0x7f7dfc13bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.949+0000 7f7dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f7dfc0ae8c0 auth_method 0 2023-08-22T23:44:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.949+0000 7f7dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.949+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.949+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 10 monclient(hunting): handle_auth_done global_id 16676 payload 274 2023-08-22T23:44:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 1 -- 172.21.15.29:0/1899678492 >> v2:172.21.15.29:3300/0 conn(0x7f7dfc0ae8c0 msgr2=0x7f7dfc13bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 1 --2- 172.21.15.29:0/1899678492 >> v2:172.21.15.29:3300/0 conn(0x7f7dfc0ae8c0 0x7f7dfc13bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 1 -- 172.21.15.29:0/1899678492 >> v1:172.21.15.52:6789/0 conn(0x7f7dfc06f5e0 legacy=0x7f7dfc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 1 monclient: found mon.noname-b 2023-08-22T23:44:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.950+0000 7f7dfbfff640 1 -- 172.21.15.29:0/1899678492 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc13fba0 con 0x7f7dfc0ad6d0 2023-08-22T23:44:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.953+0000 7f7dfbfff640 1 --2- 172.21.15.29:0/1899678492 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 0x7f7dfc13f520 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009870 tx=0x7f7dec02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92950a8a237c0bca server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 1 -- 172.21.15.29:0/1899678492 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec031640 con 0x7f7dfc0ad6d0 2023-08-22T23:44:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: dump: 2023-08-22T23:44:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:43.959 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 1 -- 172.21.15.29:0/1899678492 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec0317a0 con 0x7f7dfc0ad6d0 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 5 monclient: authenticate success, global_id 16676 2023-08-22T23:44:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 1 -- 172.21.15.29:0/1899678492 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec031b30 con 0x7f7dfc0ad6d0 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 4 set_mon_vals no callback set 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7dfaffd640 10 monclient: dump: 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:43.962 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:43.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 10 monclient: shutdown 2023-08-22T23:44:43.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:43.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1899678492 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 msgr2=0x7f7dfc13f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.957+0000 7f7e02a5e640 1 --2- 172.21.15.29:0/1899678492 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 0x7f7dfc13f520 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009870 tx=0x7f7dec02b8f0 comp rx=0 tx=0).stop 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.958+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1899678492 shutdown_connections 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.958+0000 7f7e02a5e640 1 --2- 172.21.15.29:0/1899678492 >> v2:172.21.15.29:3300/0 conn(0x7f7dfc0ae8c0 0x7f7dfc13bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.958+0000 7f7e02a5e640 1 --2- 172.21.15.29:0/1899678492 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ad6d0 0x7f7dfc13f520 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.958+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1899678492 >> 172.21.15.29:0/1899678492 conn(0x7f7dfc06a3f0 msgr2=0x7f7dfc0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.958+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1899678492 shutdown_connections 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.958+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1899678492 wait complete. 2023-08-22T23:44:43.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 10 monclient: build_initial_monmap 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 10 monclient: monmap: 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:43.966 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:43.967 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:43.967 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:43.967 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:43.967 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:43.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 1 librados: starting msgr at 2023-08-22T23:44:43.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 1 librados: starting objecter 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 1 Processor -- start 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 1 -- start start 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 1 librados: setting wanted keys 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 1 librados: calling monclient init 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.959+0000 7f7e02a5e640 10 monclient: init 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:44:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dfc0ad6d0 0x7f7dfc135540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): picked mon.noname-b con 0x7f7dfc0ad6d0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ae8c0 0x7f7dfc1c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): picked mon.noname-c con 0x7f7dfc0ae8c0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dfc0a9460 0x7f7dfc0abf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): picked mon.noname-a con 0x7f7dfc0a9460 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc1c5ab0 con 0x7f7dfc0a9460 2023-08-22T23:44:43.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc1c5090 con 0x7f7dfc0ad6d0 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0ab0b0 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.960+0000 7f7e02a5e640 10 monclient(hunting): authenticate will time out at 2848.211337s 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.963+0000 7f7dfb7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ae8c0 0x7f7dfc1c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:43.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.963+0000 7f7e00ffb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dfc0a9460 0x7f7dfc0abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:43.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7dfb7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ae8c0 0x7f7dfc1c5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34224/0 (socket says 172.21.15.29:34224) 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7dfb7fe640 1 -- 172.21.15.29:0/1081095252 learned_addr learned my addr 172.21.15.29:0/1081095252 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f7dfc0ae8c0 auth_method 0 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7e00ffb640 10 monclient(hunting): get_auth_request con 0x7f7dfc0a9460 auth_method 0 2023-08-22T23:44:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7e00ffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7e00ffb640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.964+0000 7f7e00ffb640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.965+0000 7f7dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.965+0000 7f7dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.965+0000 7f7dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.966+0000 7f7e00ffb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.966+0000 7f7e00ffb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.966+0000 7f7e00ffb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 10 monclient(hunting): handle_auth_done global_id 16679 payload 824 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 1 -- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dfc0ad6d0 msgr2=0x7f7dfc135540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dfc0ad6d0 0x7f7dfc135540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 1 -- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dfc0a9460 msgr2=0x7f7dfc0abf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dfc0a9460 0x7f7dfc0abf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 1 monclient: found mon.noname-c 2023-08-22T23:44:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.969+0000 7f7dfb7fe640 1 -- 172.21.15.29:0/1081095252 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec0094d0 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.973+0000 7f7dfb7fe640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ae8c0 0x7f7dfc1c5570 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f7df000d5c0 tx=0x7f7df000da80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.973+0000 7f7df8ff9640 10 client.?.objecter ms_handle_connect 0x7f7dfc0ae8c0 2023-08-22T23:44:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.973+0000 7f7df8ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.989+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00125f0 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.989+0000 7f7df8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: dump: 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:43.992 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: _finish_auth 0 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:43.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df0012750 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0012b00 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 5 monclient: authenticate success, global_id 16679 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 monclient: _renew_subs 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dfc0a9910 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7dfa7fc640 4 set_mon_vals no callback set 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7df8ff9640 10 monclient: dump: 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.995 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:43.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 client.16679.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 monclient: _renew_subs 2023-08-22T23:44:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dfc0ac680 con 0x7f7dfc0ae8c0 2023-08-22T23:44:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 1 librados: init done 2023-08-22T23:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 client.16679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 15 client.16679.objecter _session_command_op_assign -1 1 2023-08-22T23:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 20 client.16679.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 client.16679.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:43.990+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7dfc04dbd0 con 0x7f7dfc0ae8c0 2023-08-22T23:44:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.016+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7df0012c60 con 0x7f7dfc0ae8c0 2023-08-22T23:44:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.016+0000 7f7df8ff9640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f7de806ba30 0x7f7de806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.016+0000 7f7dfbfff640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f7de806ba30 0x7f7de806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.016+0000 7f7dfbfff640 10 monclient: get_auth_request con 0x7f7de806ba30 auth_method 0 2023-08-22T23:44:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.017+0000 7f7dfbfff640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f7de806ba30 0x7f7de806def0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7dec036000 tx=0x7f7dec030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.017+0000 7f7df8ff9640 10 client.16679.objecter ms_handle_connect 0x7f7de806ba30 2023-08-22T23:44:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f7df0058e40 con 0x7f7dfc0ae8c0 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 10 client.16679.objecter ms_dispatch 0x7f7dfc000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 3 client.16679.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 3 client.16679.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 10 client.16679.objecter checking command 1 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f7de8073230 0x7f7de80756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 20 client.16679.objecter _get_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 15 client.16679.objecter _session_command_op_remove -1 1 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 20 client.16679.objecter _get_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 20 client.16679.objecter get_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 15 client.16679.objecter _session_command_op_assign 0 1 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 4 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 10 client.16679.objecter _send_command 1 2023-08-22T23:44:44.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 --> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7de8075da0 con 0x7f7de8073230 2023-08-22T23:44:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 20 client.16679.objecter dump_active .. 0 homeless 2023-08-22T23:44:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7df0012f10 con 0x7f7dfc0ae8c0 2023-08-22T23:44:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.021+0000 7f7df8ff9640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:44.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.022+0000 7f7e00ffb640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f7de8073230 0x7f7de80756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.022+0000 7f7e00ffb640 10 monclient: get_auth_request con 0x7f7de8073230 auth_method 0 2023-08-22T23:44:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.025+0000 7f7df8ff9640 10 client.16679.objecter ms_handle_connect 0x7f7de8073230 2023-08-22T23:44:44.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.025+0000 7f7e00ffb640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f7de8073230 0x7f7de80756d0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.063+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== osd.0 v2:172.21.15.29:6808/1473840652 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f7de8075da0 con 0x7f7de8073230 2023-08-22T23:44:44.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.063+0000 7f7df8ff9640 10 client.16679.objecter ms_dispatch 0x7f7dfc000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:44.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.063+0000 7f7df8ff9640 10 client.16679.objecter _finish_command 1 = system:0 2023-08-22T23:44:44.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.063+0000 7f7df8ff9640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 4 2023-08-22T23:44:44.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.063+0000 7f7df8ff9640 15 client.16679.objecter _session_command_op_remove 0 1 2023-08-22T23:44:44.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 10 client.16679.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:44.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 15 client.16679.objecter _session_command_op_assign -1 2 2023-08-22T23:44:44.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 20 client.16679.objecter _get_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 20 client.16679.objecter _get_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 15 client.16679.objecter _session_command_op_remove -1 2 2023-08-22T23:44:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 20 client.16679.objecter get_session s=0x7f7de8072b60 osd=0 3 2023-08-22T23:44:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 15 client.16679.objecter _session_command_op_assign 0 2 2023-08-22T23:44:44.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 4 2023-08-22T23:44:44.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 10 client.16679.objecter _send_command 2 2023-08-22T23:44:44.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.090+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 --> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7dfc066290 con 0x7f7de8073230 2023-08-22T23:44:44.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7df8ff9640 1 -- 172.21.15.29:0/1081095252 <== osd.0 v2:172.21.15.29:6808/1473840652 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7dfc066290 con 0x7f7de8073230 2023-08-22T23:44:44.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7df8ff9640 10 client.16679.objecter ms_dispatch 0x7f7dfc000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:44.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7df8ff9640 10 client.16679.objecter _finish_command 2 = system:0 2023-08-22T23:44:44.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7df8ff9640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 4 2023-08-22T23:44:44.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7df8ff9640 15 client.16679.objecter _session_command_op_remove 0 2 2023-08-22T23:44:44.108 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:44.108 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:44.108 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:44.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 10 librados: watch_flush enter 2023-08-22T23:44:44.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 10 librados: watch_flush exit 2023-08-22T23:44:44.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 10 client.16679.objecter close_session for osd.0 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f7de8073230 msgr2=0x7f7de80756d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:6808/1473840652,v1:172.21.15.29:6809/1473840652] conn(0x7f7de8073230 0x7f7de80756d0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 20 client.16679.objecter put_session s=0x7f7de8072b60 osd=0 1 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 20 client.16679.objecter shutdown clearing up homeless session... 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 10 client.16679.objecter successfully canceled tick 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f7de806ba30 msgr2=0x7f7de806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f7de806ba30 0x7f7de806def0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7dec036000 tx=0x7f7dec030000 comp rx=0 tx=0).stop 2023-08-22T23:44:44.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 10 monclient: shutdown 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ae8c0 msgr2=0x7f7dfc1c5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 --2- 172.21.15.29:0/1081095252 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f7dfc0ae8c0 0x7f7dfc1c5570 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f7df000d5c0 tx=0x7f7df000da80 comp rx=0 tx=0).stop 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e00ffb640 1 -- 172.21.15.29:0/1081095252 reap_dead start 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 shutdown_connections 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 >> 172.21.15.29:0/1081095252 conn(0x7f7dfc06f5e0 msgr2=0x7f7dfc138840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 shutdown_connections 2023-08-22T23:44:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.106+0000 7f7e02a5e640 1 -- 172.21.15.29:0/1081095252 wait complete. 2023-08-22T23:44:44.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.107+0000 7f7e02a5e640 1 librados: shutdown 2023-08-22T23:44:44.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-08-22T23:44:44.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.228+0000 7f732d2b7640 10 monclient: get_monmap_and_config 2023-08-22T23:44:44.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.228+0000 7f732d2b7640 10 monclient: build_initial_monmap 2023-08-22T23:44:44.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.228+0000 7f732d2b7640 10 monclient: monmap: 2023-08-22T23:44:44.229 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:44.229 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:44.229422+0000 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:44.229422+0000 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:44.230 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:44.231 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:44.231 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:44.231 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.228+0000 7f732d2b7640 1 Processor -- start 2023-08-22T23:44:44.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.228+0000 7f732d2b7640 1 -- start start 2023-08-22T23:44:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.228+0000 7f732d2b7640 10 monclient: init 2023-08-22T23:44:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-08-22T23:44:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f73281448e0 0x7f7328144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): picked mon.noname-c con 0x7f73281448e0 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73280add90 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f732813e8a0 0x7f7328145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): picked mon.noname-a con 0x7f732813e8a0 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73280b12d0 con 0x7f73280add90 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73280afae0 con 0x7f732813e8a0 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73280afdc0 con 0x7f73281448e0 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.229+0000 7f732d2b7640 10 monclient(hunting): authenticate will time out at 2848.480297s 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.230+0000 7f7326ffd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f73281448e0 0x7f7328144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f73281448e0 0x7f7328144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:33214/0 (socket says 172.21.15.29:33214) 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 1 -- 172.21.15.29:0/3510719279 learned_addr learned my addr 172.21.15.29:0/3510719279 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): get_auth_request con 0x7f73281448e0 auth_method 0 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.231+0000 7f7326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 10 monclient(hunting): handle_auth_done global_id 19567 payload 274 2023-08-22T23:44:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 1 -- 172.21.15.29:0/3510719279 >> v2:172.21.15.29:3300/0 conn(0x7f732813e8a0 msgr2=0x7f7328145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 1 --2- 172.21.15.29:0/3510719279 >> v2:172.21.15.29:3300/0 conn(0x7f732813e8a0 0x7f7328145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 1 -- 172.21.15.29:0/3510719279 >> v1:172.21.15.29:6789/0 conn(0x7f73280add90 legacy=0x7f732813e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:44:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 1 monclient: found mon.noname-c 2023-08-22T23:44:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.233+0000 7f7326ffd640 1 -- 172.21.15.29:0/3510719279 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73280b00a0 con 0x7f73281448e0 2023-08-22T23:44:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.235+0000 7f7326ffd640 1 --2- 172.21.15.29:0/3510719279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73281448e0 0x7f7328144cc0 secure :-1 s=READY pgs=4623 cs=0 l=1 rev1=1 crypto rx=0x7f73280b7890 tx=0x7f731802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e71515709620d25 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 1 -- 172.21.15.29:0/3510719279 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73180317e0 con 0x7f73281448e0 2023-08-22T23:44:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: dump: 2023-08-22T23:44:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.240 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: _finish_auth 0 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 1 -- 172.21.15.29:0/3510719279 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7318031940 con 0x7f73281448e0 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 1 -- 172.21.15.29:0/3510719279 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318031cd0 con 0x7f73281448e0 2023-08-22T23:44:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.238+0000 7f7325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f7325ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f7325ffb640 10 monclient: dump: 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 5 monclient: authenticate success, global_id 19567 2023-08-22T23:44:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 4 set_mon_vals no callback set 2023-08-22T23:44:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 10 monclient: shutdown 2023-08-22T23:44:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 1 -- 172.21.15.29:0/3510719279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73281448e0 msgr2=0x7f7328144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 1 --2- 172.21.15.29:0/3510719279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73281448e0 0x7f7328144cc0 secure :-1 s=READY pgs=4623 cs=0 l=1 rev1=1 crypto rx=0x7f73280b7890 tx=0x7f731802b880 comp rx=0 tx=0).stop 2023-08-22T23:44:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 1 -- 172.21.15.29:0/3510719279 shutdown_connections 2023-08-22T23:44:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 1 --2- 172.21.15.29:0/3510719279 >> v2:172.21.15.29:3300/0 conn(0x7f732813e8a0 0x7f7328145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.239+0000 7f732d2b7640 1 --2- 172.21.15.29:0/3510719279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73281448e0 0x7f7328144cc0 unknown :-1 s=CLOSED pgs=4623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.240+0000 7f732d2b7640 1 -- 172.21.15.29:0/3510719279 >> 172.21.15.29:0/3510719279 conn(0x7f7328065ed0 msgr2=0x7f73280b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.240+0000 7f732d2b7640 1 -- 172.21.15.29:0/3510719279 shutdown_connections 2023-08-22T23:44:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.240+0000 7f732d2b7640 1 -- 172.21.15.29:0/3510719279 wait complete. 2023-08-22T23:44:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 10 monclient: build_initial_monmap 2023-08-22T23:44:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 10 monclient: monmap: 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 1 librados: starting msgr at 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 1 librados: starting objecter 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 1 Processor -- start 2023-08-22T23:44:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 1 -- start start 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 1 librados: setting wanted keys 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 1 librados: calling monclient init 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.241+0000 7f732d2b7640 10 monclient: init 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 0x7f7328143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): picked mon.noname-b con 0x7f73280add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f732813e8a0 0x7f732813ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): picked mon.noname-c con 0x7f732813e8a0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73281448e0 0x7f73281d9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): picked mon.noname-a con 0x7f73281448e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7328143e00 con 0x7f73281448e0 2023-08-22T23:44:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732813b510 con 0x7f73280add90 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7328134c00 con 0x7f732813e8a0 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.242+0000 7f732d2b7640 10 monclient(hunting): authenticate will time out at 2848.493281s 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 0x7f7328143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 0x7f7328143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:33224/0 (socket says 172.21.15.29:33224) 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 1 -- 172.21.15.29:0/1360326113 learned_addr learned my addr 172.21.15.29:0/1360326113 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 10 monclient(hunting): get_auth_request con 0x7f73280add90 auth_method 0 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.243+0000 7f7326ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.244+0000 7f7326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.244+0000 7f7326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.244+0000 7f7326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.246+0000 7f73267fc640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f732813e8a0 0x7f732813ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 10 monclient(hunting): handle_auth_done global_id 19570 payload 824 2023-08-22T23:44:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 1 -- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f732813e8a0 msgr2=0x7f732813ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f732813e8a0 0x7f732813ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 1 -- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73281448e0 msgr2=0x7f73281d9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73281448e0 0x7f73281d9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 1 monclient: found mon.noname-b 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.247+0000 7f7326ffd640 1 -- 172.21.15.29:0/1360326113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73180094d0 con 0x7f73280add90 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.249+0000 7f7326ffd640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 0x7f7328143730 secure :-1 s=READY pgs=4624 cs=0 l=1 rev1=1 crypto rx=0x7f731802b8c0 tx=0x7f73180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.249+0000 7f730f7fe640 10 client.?.objecter ms_handle_connect 0x7f73280add90 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.249+0000 7f730f7fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318032940 con 0x7f73280add90 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: dump: 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: _finish_auth 0 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:44:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7318032aa0 con 0x7f73280add90 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f732d2b7640 5 monclient: authenticate success, global_id 19570 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f732d2b7640 10 monclient: _renew_subs 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f732d2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7328134e20 con 0x7f73280add90 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:44.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318032e50 con 0x7f73280add90 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f730f7fe640 10 monclient: dump: 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.252+0000 7f73257fa640 4 set_mon_vals no callback set 2023-08-22T23:44:44.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 10 client.19570.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 10 monclient: _renew_subs 2023-08-22T23:44:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73281d9c40 con 0x7f73280add90 2023-08-22T23:44:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 1 librados: init done 2023-08-22T23:44:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:44.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:44.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.253+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732813b6d0 con 0x7f73280add90 2023-08-22T23:44:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.274+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f731804e450 con 0x7f73280add90 2023-08-22T23:44:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.275+0000 7f730f7fe640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f731406ba60 0x7f731406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.275+0000 7f73267fc640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f731406ba60 0x7f731406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.275+0000 7f73267fc640 10 monclient: get_auth_request con 0x7f731406ba60 auth_method 0 2023-08-22T23:44:44.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.275+0000 7f73267fc640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f731406ba60 0x7f731406df20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f732813b8a0 tx=0x7f731c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.275+0000 7f730f7fe640 10 client.19570.objecter ms_handle_connect 0x7f731406ba60 2023-08-22T23:44:44.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.277+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f731807a560 con 0x7f73280add90 2023-08-22T23:44:44.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.277+0000 7f730f7fe640 10 client.19570.objecter ms_dispatch 0x7f7328000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:44.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.277+0000 7f730f7fe640 3 client.19570.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:44.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.277+0000 7f730f7fe640 3 client.19570.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:44.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.277+0000 7f730f7fe640 20 client.19570.objecter dump_active .. 0 homeless 2023-08-22T23:44:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.528+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f7318076210 con 0x7f73280add90 2023-08-22T23:44:44.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.528+0000 7f730f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.528+0000 7f730f7fe640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.728+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f73280ae200 con 0x7f731406ba60 2023-08-22T23:44:44.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.729+0000 7f730f7fe640 1 -- 172.21.15.29:0/1360326113 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f731c0050d0 con 0x7f731406ba60 2023-08-22T23:44:44.730 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 0 to deep-scrub 2023-08-22T23:44:44.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 10 librados: watch_flush enter 2023-08-22T23:44:44.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 10 librados: watch_flush exit 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 20 client.19570.objecter shutdown clearing up homeless session... 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 10 client.19570.objecter successfully canceled tick 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f731406ba60 msgr2=0x7f731406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f731406ba60 0x7f731406df20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f732813b8a0 tx=0x7f731c009040 comp rx=0 tx=0).stop 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 10 monclient: shutdown 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 msgr2=0x7f7328143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 0x7f7328143730 secure :-1 s=READY pgs=4624 cs=0 l=1 rev1=1 crypto rx=0x7f731802b8c0 tx=0x7f73180046c0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 shutdown_connections 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f731406ba60 0x7f731406df20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73281448e0 0x7f73281d9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f732813e8a0 0x7f732813ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 --2- 172.21.15.29:0/1360326113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73280add90 0x7f7328143730 unknown :-1 s=CLOSED pgs=4624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 >> 172.21.15.29:0/1360326113 conn(0x7f7328065ed0 msgr2=0x7f7328137fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 shutdown_connections 2023-08-22T23:44:44.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 -- 172.21.15.29:0/1360326113 wait complete. 2023-08-22T23:44:44.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.731+0000 7f732d2b7640 1 librados: shutdown 2023-08-22T23:44:44.743 INFO:tasks.ceph:Scrubbing osd.1 2023-08-22T23:44:44.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:44.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.852+0000 7f9910c09640 10 monclient: get_monmap_and_config 2023-08-22T23:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.852+0000 7f9910c09640 10 monclient: build_initial_monmap 2023-08-22T23:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.852+0000 7f9910c09640 10 monclient: monmap: 2023-08-22T23:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:44.854144+0000 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:44.854144+0000 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:44.856 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:44.856 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:44.856 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.853+0000 7f9910c09640 1 Processor -- start 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.853+0000 7f9910c09640 1 -- start start 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.853+0000 7f9910c09640 10 monclient: init 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f990c0a7950 0x7f990c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): picked mon.noname-c con 0x7f990c0a7950 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:44.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f990c0b0520 0x7f990c141100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): picked mon.noname-a con 0x7f990c0b0520 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f990c141640 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990c0a8b70 con 0x7f990c141640 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990c0a9930 con 0x7f990c0b0520 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990c0a9c10 con 0x7f990c0a7950 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.854+0000 7f9910c09640 10 monclient(hunting): authenticate will time out at 2849.104765s 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f990c0a7950 0x7f990c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f9909d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f990c0b0520 0x7f990c141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f990c141640 0x7f990c144680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48092/0 (socket says 172.21.15.29:48092) 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990ad9d640 1 -- 172.21.15.29:0/3173067921 learned_addr learned my addr 172.21.15.29:0/3173067921 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): get_auth_request con 0x7f990c0a7950 auth_method 0 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f990a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f9909d9b640 10 monclient(hunting): get_auth_request con 0x7f990c0b0520 auth_method 0 2023-08-22T23:44:44.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f9909d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f9909d9b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.855+0000 7f9909d9b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.856+0000 7f9909d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.856+0000 7f9909d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.856+0000 7f9909d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 10 monclient(hunting): handle_auth_done global_id 19573 payload 274 2023-08-22T23:44:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 1 -- 172.21.15.29:0/3173067921 >> v2:172.21.15.29:3300/0 conn(0x7f990c0b0520 msgr2=0x7f990c141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 1 --2- 172.21.15.29:0/3173067921 >> v2:172.21.15.29:3300/0 conn(0x7f990c0b0520 0x7f990c141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 1 -- 172.21.15.29:0/3173067921 >> v1:172.21.15.29:6789/0 conn(0x7f990c141640 legacy=0x7f990c144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 10 monclient: _finish_hunting 0 2023-08-22T23:44:44.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 1 monclient: found mon.noname-c 2023-08-22T23:44:44.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:44.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:44.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.857+0000 7f990a59c640 1 -- 172.21.15.29:0/3173067921 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c0a9ef0 con 0x7f990c0a7950 2023-08-22T23:44:44.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.859+0000 7f990a59c640 1 --2- 172.21.15.29:0/3173067921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0a7950 0x7f990c0affe0 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 crypto rx=0x7f990c066480 tx=0x7f990402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21e5fdc77d59183d server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 1 -- 172.21.15.29:0/3173067921 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904031830 con 0x7f990c0a7950 2023-08-22T23:44:44.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: dump: 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.866 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: _finish_auth 0 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: _check_auth_tickets 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:44.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 1 -- 172.21.15.29:0/3173067921 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904031990 con 0x7f990c0a7950 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 1 -- 172.21.15.29:0/3173067921 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904031d20 con 0x7f990c0a7950 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f990959a640 10 monclient: dump: 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 5 monclient: authenticate success, global_id 19573 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 4 set_mon_vals no callback set 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 10 monclient: shutdown 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 1 -- 172.21.15.29:0/3173067921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0a7950 msgr2=0x7f990c0affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.863+0000 7f9910c09640 1 --2- 172.21.15.29:0/3173067921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0a7950 0x7f990c0affe0 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 crypto rx=0x7f990c066480 tx=0x7f990402b8d0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 1 -- 172.21.15.29:0/3173067921 shutdown_connections 2023-08-22T23:44:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 1 --2- 172.21.15.29:0/3173067921 >> v2:172.21.15.29:3300/0 conn(0x7f990c0b0520 0x7f990c141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 1 --2- 172.21.15.29:0/3173067921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0a7950 0x7f990c0affe0 unknown :-1 s=CLOSED pgs=4625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 1 -- 172.21.15.29:0/3173067921 >> 172.21.15.29:0/3173067921 conn(0x7f990c065f70 msgr2=0x7f990c0b6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 1 -- 172.21.15.29:0/3173067921 shutdown_connections 2023-08-22T23:44:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 1 -- 172.21.15.29:0/3173067921 wait complete. 2023-08-22T23:44:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 10 monclient: build_initial_monmap 2023-08-22T23:44:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.864+0000 7f9910c09640 10 monclient: monmap: 2023-08-22T23:44:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 librados: starting msgr at 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 librados: starting objecter 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 Processor -- start 2023-08-22T23:44:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 -- start start 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 librados: setting wanted keys 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 librados: calling monclient init 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 10 monclient: init 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f990c0a7950 0x7f990c1408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 10 monclient(hunting): picked mon.noname-c con 0x7f990c0a7950 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0b0520 0x7f990c0aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 10 monclient(hunting): picked mon.noname-b con 0x7f990c0b0520 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f990c141640 0x7f990c143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.865+0000 7f9910c09640 10 monclient(hunting): picked mon.noname-a con 0x7f990c141640 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c0ad3f0 con 0x7f990c141640 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c143070 con 0x7f990c0b0520 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c0ac990 con 0x7f990c0a7950 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9910c09640 10 monclient(hunting): authenticate will time out at 2849.116492s 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f990c141640 0x7f990c143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990a59c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f990c0a7950 0x7f990c1408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f990c141640 0x7f990c143770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52632/0 (socket says 172.21.15.29:52632) 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 1 -- 172.21.15.29:0/1083817795 learned_addr learned my addr 172.21.15.29:0/1083817795 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 10 monclient(hunting): get_auth_request con 0x7f990c141640 auth_method 0 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f9909d9b640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0b0520 0x7f990c0aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990a59c640 10 monclient(hunting): get_auth_request con 0x7f990c0a7950 auth_method 0 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990a59c640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.866+0000 7f990a59c640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): get_auth_request con 0x7f990c0b0520 auth_method 0 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f990ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f990ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f990ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f990a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f990a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f990a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.867+0000 7f9909d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 10 monclient(hunting): handle_auth_done global_id 14763 payload 824 2023-08-22T23:44:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 1 -- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f990c0a7950 msgr2=0x7f990c1408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f990c0a7950 0x7f990c1408a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 1 -- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0b0520 msgr2=0x7f990c0aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f990c0b0520 0x7f990c0aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 10 monclient: _finish_hunting 0 2023-08-22T23:44:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 1 monclient: found mon.noname-a 2023-08-22T23:44:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.870+0000 7f990ad9d640 1 -- 172.21.15.29:0/1083817795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040093a0 con 0x7f990c141640 2023-08-22T23:44:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.872+0000 7f990ad9d640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f990c141640 0x7f990c143770 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00cbc0 tx=0x7f98fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.872+0000 7f9902ffd640 10 client.?.objecter ms_handle_connect 0x7f990c141640 2023-08-22T23:44:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.872+0000 7f9902ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00ad40 con 0x7f990c141640 2023-08-22T23:44:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: dump: 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.887 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc012400 con 0x7f990c141640 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00aea0 con 0x7f990c141640 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 5 monclient: authenticate success, global_id 14763 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 10 monclient: _renew_subs 2023-08-22T23:44:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c143cb0 con 0x7f990c141640 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9908d99640 4 set_mon_vals no callback set 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9902ffd640 10 monclient: dump: 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 10 client.14763.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 10 monclient: _renew_subs 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c0ac1d0 con 0x7f990c141640 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.876+0000 7f9910c09640 1 librados: init done 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.877+0000 7f9910c09640 10 client.14763.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.877+0000 7f9910c09640 15 client.14763.objecter _session_command_op_assign -1 1 2023-08-22T23:44:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.877+0000 7f9910c09640 20 client.14763.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:44.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.877+0000 7f9910c09640 10 client.14763.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:44.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.877+0000 7f9910c09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:44.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.877+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f990c141aa0 con 0x7f990c141640 2023-08-22T23:44:44.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.896+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f98fc007070 con 0x7f990c141640 2023-08-22T23:44:44.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.896+0000 7f9902ffd640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f98ec06ba30 0x7f98ec06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.897+0000 7f990a59c640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f98ec06ba30 0x7f98ec06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.897+0000 7f990a59c640 10 monclient: get_auth_request con 0x7f98ec06ba30 auth_method 0 2023-08-22T23:44:44.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.897+0000 7f990a59c640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f98ec06ba30 0x7f98ec06def0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9904036000 tx=0x7f990403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.897+0000 7f9902ffd640 10 client.14763.objecter ms_handle_connect 0x7f98ec06ba30 2023-08-22T23:44:44.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f98fc05bdd0 con 0x7f990c141640 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 10 client.14763.objecter ms_dispatch 0x7f990c000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 3 client.14763.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 3 client.14763.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 10 client.14763.objecter checking command 1 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7f98ec073230 0x7f98ec0756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 20 client.14763.objecter _get_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 15 client.14763.objecter _session_command_op_remove -1 1 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 20 client.14763.objecter _get_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 20 client.14763.objecter get_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 15 client.14763.objecter _session_command_op_assign 1 1 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 4 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 10 client.14763.objecter _send_command 1 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 --> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98ec075da0 con 0x7f98ec073230 2023-08-22T23:44:44.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.898+0000 7f9902ffd640 20 client.14763.objecter dump_active .. 0 homeless 2023-08-22T23:44:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.899+0000 7f9909d9b640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7f98ec073230 0x7f98ec0756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.899+0000 7f9909d9b640 10 monclient: get_auth_request con 0x7f98ec073230 auth_method 0 2023-08-22T23:44:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.900+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98fc054400 con 0x7f990c141640 2023-08-22T23:44:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.900+0000 7f9902ffd640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:44.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.902+0000 7f9909d9b640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7f98ec073230 0x7f98ec0756d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:44.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.902+0000 7f9902ffd640 10 client.14763.objecter ms_handle_connect 0x7f98ec073230 2023-08-22T23:44:44.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.926+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== osd.1 v2:172.21.15.29:6824/2619884743 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f98ec075da0 con 0x7f98ec073230 2023-08-22T23:44:44.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.926+0000 7f9902ffd640 10 client.14763.objecter ms_dispatch 0x7f990c000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:44.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.926+0000 7f9902ffd640 10 client.14763.objecter _finish_command 1 = system:0 2023-08-22T23:44:44.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.926+0000 7f9902ffd640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 4 2023-08-22T23:44:44.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.926+0000 7f9902ffd640 15 client.14763.objecter _session_command_op_remove 1 1 2023-08-22T23:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 10 client.14763.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 15 client.14763.objecter _session_command_op_assign -1 2 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 20 client.14763.objecter _get_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 20 client.14763.objecter _get_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 15 client.14763.objecter _session_command_op_remove -1 2 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 20 client.14763.objecter get_session s=0x7f98ec072b60 osd=1 3 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 15 client.14763.objecter _session_command_op_assign 1 2 2023-08-22T23:44:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 4 2023-08-22T23:44:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 10 client.14763.objecter _send_command 2 2023-08-22T23:44:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.939+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 --> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f990c0a8600 con 0x7f98ec073230 2023-08-22T23:44:44.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.953+0000 7f9902ffd640 1 -- 172.21.15.29:0/1083817795 <== osd.1 v2:172.21.15.29:6824/2619884743 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f98f4009d50 con 0x7f98ec073230 2023-08-22T23:44:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.953+0000 7f9902ffd640 10 client.14763.objecter ms_dispatch 0x7f990c000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.953+0000 7f9902ffd640 10 client.14763.objecter _finish_command 2 = system:0 2023-08-22T23:44:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.953+0000 7f9902ffd640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 4 2023-08-22T23:44:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.953+0000 7f9902ffd640 15 client.14763.objecter _session_command_op_remove 1 2 2023-08-22T23:44:44.955 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:44.956 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 10 librados: watch_flush enter 2023-08-22T23:44:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 10 librados: watch_flush exit 2023-08-22T23:44:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 10 client.14763.objecter close_session for osd.1 2023-08-22T23:44:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7f98ec073230 msgr2=0x7f98ec0756d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:6824/2619884743,v1:172.21.15.29:6825/2619884743] conn(0x7f98ec073230 0x7f98ec0756d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 20 client.14763.objecter put_session s=0x7f98ec072b60 osd=1 1 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 20 client.14763.objecter shutdown clearing up homeless session... 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 10 client.14763.objecter successfully canceled tick 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f98ec06ba30 msgr2=0x7f98ec06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f98ec06ba30 0x7f98ec06def0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9904036000 tx=0x7f990403c040 comp rx=0 tx=0).stop 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 10 monclient: shutdown 2023-08-22T23:44:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f990c141640 msgr2=0x7f990c143770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 --2- 172.21.15.29:0/1083817795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f990c141640 0x7f990c143770 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00cbc0 tx=0x7f98fc00a430 comp rx=0 tx=0).stop 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f990ad9d640 1 -- 172.21.15.29:0/1083817795 reap_dead start 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 shutdown_connections 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 >> 172.21.15.29:0/1083817795 conn(0x7f990c065f70 msgr2=0x7f990c13c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 shutdown_connections 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 -- 172.21.15.29:0/1083817795 wait complete. 2023-08-22T23:44:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:44.954+0000 7f9910c09640 1 librados: shutdown 2023-08-22T23:44:44.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-08-22T23:44:45.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.070+0000 7f3a5c057640 10 monclient: get_monmap_and_config 2023-08-22T23:44:45.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.070+0000 7f3a5c057640 10 monclient: build_initial_monmap 2023-08-22T23:44:45.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.070+0000 7f3a5c057640 10 monclient: monmap: 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:45.072264+0000 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:45.072264+0000 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.073 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:45.074 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:45.074 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:45.074 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:45.074 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:45.074 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.071+0000 7f3a5c057640 1 Processor -- start 2023-08-22T23:44:45.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.071+0000 7f3a5c057640 1 -- start start 2023-08-22T23:44:45.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.071+0000 7f3a5c057640 10 monclient: init 2023-08-22T23:44:45.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-08-22T23:44:45.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a540a7d50 0x7f3a540a8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): picked mon.noname-a con 0x7f3a540a7d50 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:45.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 0x7f3a541411e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): picked mon.noname-b con 0x7f3a540aff90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a540b03b0 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5413ee50 con 0x7f3a540b03b0 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a540a87c0 con 0x7f3a540a7d50 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540a9bf0 con 0x7f3a540aff90 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a5c057640 10 monclient(hunting): authenticate will time out at 2849.322935s 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.072+0000 7f3a595f2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 0x7f3a541411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a59df3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a540a7d50 0x7f3a540a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 0x7f3a541411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34250/0 (socket says 172.21.15.29:34250) 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 1 -- 172.21.15.29:0/1641297676 learned_addr learned my addr 172.21.15.29:0/1641297676 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:45.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): get_auth_request con 0x7f3a540aff90 auth_method 0 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a5a5f4640 1 --1- 172.21.15.29:0/1641297676 >> v1:172.21.15.29:6789/0 conn(0x7f3a540b03b0 0x7f3a54141720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48094/0 (socket says 172.21.15.29:48094) 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a59df3640 10 monclient(hunting): get_auth_request con 0x7f3a540a7d50 auth_method 0 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a59df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a59df3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a59df3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.073+0000 7f3a595f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.074+0000 7f3a59df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.074+0000 7f3a59df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.074+0000 7f3a59df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 10 monclient(hunting): handle_auth_done global_id 16685 payload 274 2023-08-22T23:44:45.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 1 -- 172.21.15.29:0/1641297676 >> v2:172.21.15.29:3300/0 conn(0x7f3a540a7d50 msgr2=0x7f3a540a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:45.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 1 --2- 172.21.15.29:0/1641297676 >> v2:172.21.15.29:3300/0 conn(0x7f3a540a7d50 0x7f3a540a8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 1 -- 172.21.15.29:0/1641297676 >> v1:172.21.15.29:6789/0 conn(0x7f3a540b03b0 legacy=0x7f3a54141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 1 monclient: found mon.noname-b 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.075+0000 7f3a595f2640 1 -- 172.21.15.29:0/1641297676 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540a9df0 con 0x7f3a540aff90 2023-08-22T23:44:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.077+0000 7f3a595f2640 1 --2- 172.21.15.29:0/1641297676 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 0x7f3a541411e0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009870 tx=0x7f3a4402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a321e72f845f0a server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 1 -- 172.21.15.29:0/1641297676 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440317c0 con 0x7f3a540aff90 2023-08-22T23:44:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: dump: 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 1 -- 172.21.15.29:0/1641297676 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44031920 con 0x7f3a540aff90 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a5c057640 5 monclient: authenticate success, global_id 16685 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a5c057640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 1 -- 172.21.15.29:0/1641297676 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44031cb0 con 0x7f3a540aff90 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a5c057640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a5c057640 4 set_mon_vals no callback set 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.081+0000 7f3a58df1640 10 monclient: dump: 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 10 monclient: shutdown 2023-08-22T23:44:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 -- 172.21.15.29:0/1641297676 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 msgr2=0x7f3a541411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 --2- 172.21.15.29:0/1641297676 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 0x7f3a541411e0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009870 tx=0x7f3a4402b860 comp rx=0 tx=0).stop 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 -- 172.21.15.29:0/1641297676 shutdown_connections 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 --2- 172.21.15.29:0/1641297676 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540aff90 0x7f3a541411e0 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 --2- 172.21.15.29:0/1641297676 >> v2:172.21.15.29:3300/0 conn(0x7f3a540a7d50 0x7f3a540a8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 -- 172.21.15.29:0/1641297676 >> 172.21.15.29:0/1641297676 conn(0x7f3a54065dd0 msgr2=0x7f3a540b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 -- 172.21.15.29:0/1641297676 shutdown_connections 2023-08-22T23:44:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.082+0000 7f3a5c057640 1 -- 172.21.15.29:0/1641297676 wait complete. 2023-08-22T23:44:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 10 monclient: build_initial_monmap 2023-08-22T23:44:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 10 monclient: monmap: 2023-08-22T23:44:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:45.093 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 1 librados: starting msgr at 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 1 librados: starting objecter 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 1 Processor -- start 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 1 -- start start 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 1 librados: setting wanted keys 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 1 librados: calling monclient init 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.083+0000 7f3a5c057640 10 monclient: init 2023-08-22T23:44:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 0x7f3a540ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): picked mon.noname-c con 0x7f3a540a7d50 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a540aff90 0x7f3a54120ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): picked mon.noname-b con 0x7f3a540aff90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a540b03b0 0x7f3a541d9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): picked mon.noname-a con 0x7f3a540b03b0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54121750 con 0x7f3a540b03b0 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54121be0 con 0x7f3a540aff90 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5411fbd0 con 0x7f3a540a7d50 2023-08-22T23:44:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5c057640 10 monclient(hunting): authenticate will time out at 2849.334861s 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a59df3640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 0x7f3a540ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.084+0000 7f3a5a5f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a540b03b0 0x7f3a541d9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 0x7f3a540ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34254/0 (socket says 172.21.15.29:34254) 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 1 -- 172.21.15.29:0/5802508 learned_addr learned my addr 172.21.15.29:0/5802508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): get_auth_request con 0x7f3a540a7d50 auth_method 0 2023-08-22T23:44:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): get_auth_request con 0x7f3a540b03b0 auth_method 0 2023-08-22T23:44:45.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a540aff90 0x7f3a54120ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 10 monclient(hunting): get_auth_request con 0x7f3a540aff90 auth_method 0 2023-08-22T23:44:45.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a5a5f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a59df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.085+0000 7f3a595f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.086+0000 7f3a595f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 10 monclient(hunting): handle_auth_done global_id 14772 payload 824 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 1 -- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 msgr2=0x7f3a540ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 0x7f3a540ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 1 -- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a540aff90 msgr2=0x7f3a54120ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a540aff90 0x7f3a54120ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 1 monclient: found mon.noname-a 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a5a5f4640 1 -- 172.21.15.29:0/5802508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440094d0 con 0x7f3a540b03b0 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.088+0000 7f3a59df3640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 0x7f3a540ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.090+0000 7f3a5a5f4640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a540b03b0 0x7f3a541d9350 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f3a54141e30 tx=0x7f3a4c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.090+0000 7f3a4a7fc640 10 client.?.objecter ms_handle_connect 0x7f3a540b03b0 2023-08-22T23:44:45.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.090+0000 7f3a4a7fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c004d60 con 0x7f3a540b03b0 2023-08-22T23:44:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: dump: 2023-08-22T23:44:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.107 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c00a640 con 0x7f3a540b03b0 2023-08-22T23:44:45.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:45.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c00aa10 con 0x7f3a540b03b0 2023-08-22T23:44:45.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4a7fc640 10 monclient: dump: 2023-08-22T23:44:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 5 monclient: authenticate success, global_id 14772 2023-08-22T23:44:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 10 monclient: _renew_subs 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a54120cd0 con 0x7f3a540b03b0 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a4b7fe640 4 set_mon_vals no callback set 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 10 client.14772.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 10 monclient: _renew_subs 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a541d9a70 con 0x7f3a540b03b0 2023-08-22T23:44:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.094+0000 7f3a5c057640 1 librados: init done 2023-08-22T23:44:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.095+0000 7f3a5c057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.095+0000 7f3a5c057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.095+0000 7f3a5c057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.095+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a54121530 con 0x7f3a540b03b0 2023-08-22T23:44:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.353+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3a4c00ab70 con 0x7f3a540b03b0 2023-08-22T23:44:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.353+0000 7f3a4a7fc640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806b9e0 0x7f3a3806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.354+0000 7f3a59df3640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806b9e0 0x7f3a3806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.354+0000 7f3a59df3640 10 monclient: get_auth_request con 0x7f3a3806b9e0 auth_method 0 2023-08-22T23:44:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.354+0000 7f3a59df3640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806b9e0 0x7f3a3806dea0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3a500112c0 tx=0x7f3a50009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.354+0000 7f3a4a7fc640 10 client.14772.objecter ms_handle_connect 0x7f3a3806b9e0 2023-08-22T23:44:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f3a4c00ae20 con 0x7f3a540b03b0 2023-08-22T23:44:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 10 client.14772.objecter ms_dispatch 0x7f3a54000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 3 client.14772.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 3 client.14772.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 20 client.14772.objecter dump_active .. 0 homeless 2023-08-22T23:44:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f3a4c053400 con 0x7f3a540b03b0 2023-08-22T23:44:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:45.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.366+0000 7f3a4a7fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:45.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.541+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a540ad6e0 con 0x7f3a3806b9e0 2023-08-22T23:44:45.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.542+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/5802508 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3a50009d20 con 0x7f3a3806b9e0 2023-08-22T23:44:45.543 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 1 to deep-scrub 2023-08-22T23:44:45.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 10 librados: watch_flush enter 2023-08-22T23:44:45.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 10 librados: watch_flush exit 2023-08-22T23:44:45.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 20 client.14772.objecter shutdown clearing up homeless session... 2023-08-22T23:44:45.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 10 client.14772.objecter successfully canceled tick 2023-08-22T23:44:45.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806b9e0 msgr2=0x7f3a3806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806b9e0 0x7f3a3806dea0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3a500112c0 tx=0x7f3a50009190 comp rx=0 tx=0).stop 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 10 monclient: shutdown 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a540b03b0 msgr2=0x7f3a541d9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a540b03b0 0x7f3a541d9350 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f3a54141e30 tx=0x7f3a4c00cb90 comp rx=0 tx=0).stop 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 shutdown_connections 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806b9e0 0x7f3a3806dea0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a540b03b0 0x7f3a541d9350 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a540aff90 0x7f3a54120ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 --2- 172.21.15.29:0/5802508 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a540a7d50 0x7f3a540ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 >> 172.21.15.29:0/5802508 conn(0x7f3a54065dd0 msgr2=0x7f3a540b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 shutdown_connections 2023-08-22T23:44:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.544+0000 7f3a5c057640 1 -- 172.21.15.29:0/5802508 wait complete. 2023-08-22T23:44:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.545+0000 7f3a5c057640 1 librados: shutdown 2023-08-22T23:44:45.558 INFO:tasks.ceph:Scrubbing osd.2 2023-08-22T23:44:45.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:45.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.671+0000 7f9b09b38640 10 monclient: get_monmap_and_config 2023-08-22T23:44:45.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.671+0000 7f9b09b38640 10 monclient: build_initial_monmap 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.671+0000 7f9b09b38640 10 monclient: monmap: 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:45.672772+0000 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:45.672772+0000 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.674 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:45.674 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:45.674 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:45.674 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:45.674 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:45.674 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 1 Processor -- start 2023-08-22T23:44:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 1 -- start start 2023-08-22T23:44:45.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 10 monclient: init 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f9b040a81b0 0x7f9b040a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 10 monclient(hunting): picked mon.noname-c con 0x7f9b040a81b0 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.672+0000 7f9b09b38640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b0413e760 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b04144e30 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b040b07e0 con 0x7f9b04144e30 2023-08-22T23:44:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b040a98a0 con 0x7f9b0413e760 2023-08-22T23:44:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b040a7d70 con 0x7f9b040a81b0 2023-08-22T23:44:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.673+0000 7f9b09b38640 10 monclient(hunting): authenticate will time out at 2849.923537s 2023-08-22T23:44:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.700+0000 7f9b03fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b04144e30 0x7f9b0414ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48096/0 (socket says 172.21.15.29:48096) 2023-08-22T23:44:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.700+0000 7f9b03fff640 1 -- 172.21.15.29:0/1295064232 learned_addr learned my addr 172.21.15.29:0/1295064232 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:45.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.703+0000 7f9b037fe640 1 --2- 172.21.15.29:0/1295064232 >> v2:172.21.15.52:3300/0 conn(0x7f9b040a81b0 0x7f9b040a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.704+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307104750 0 0) 0x7f9b040b07e0 con 0x7f9b04144e30 2023-08-22T23:44:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.704+0000 7f9b027fc640 10 monclient(hunting): _negotiate 2023-08-22T23:44:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.704+0000 7f9b027fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.704+0000 7f9b027fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.704+0000 7f9b027fc640 10 monclient(hunting): my global_id is 14775 2023-08-22T23:44:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.704+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aec003520 con 0x7f9b04144e30 2023-08-22T23:44:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.707+0000 7f9b037fe640 10 monclient(hunting): get_auth_request con 0x7f9b040a81b0 auth_method 0 2023-08-22T23:44:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.707+0000 7f9b037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.707+0000 7f9b037fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.707+0000 7f9b037fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.708+0000 7f9b037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.708+0000 7f9b037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.708+0000 7f9b037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1204537328 0 0) 0x7f9aec003520 con 0x7f9b04144e30 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 >> v2:172.21.15.52:3300/0 conn(0x7f9b040a81b0 msgr2=0x7f9b040a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 --2- 172.21.15.29:0/1295064232 >> v2:172.21.15.52:3300/0 conn(0x7f9b040a81b0 0x7f9b040a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 >> v1:172.21.15.52:6789/0 conn(0x7f9b0413e760 legacy=0x7f9b04144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 monclient: found mon.noname-a-legacy 2023-08-22T23:44:45.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.29:6789/0 2023-08-22T23:44:45.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040a8ad0 con 0x7f9b04144e30 2023-08-22T23:44:45.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4066382038 0 0) 0x7f9afc001670 con 0x7f9b04144e30 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b09b38640 5 monclient: authenticate success, global_id 14775 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b09b38640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: dump: 2023-08-22T23:44:45.714 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.715 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.716 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] but i'm connected to v1:172.21.15.29:6789/0, reconnecting 2023-08-22T23:44:45.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 >> v1:172.21.15.29:6789/0 conn(0x7f9b04144e30 legacy=0x7f9b0414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec003970 0x7f9aec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): picked mon.a con 0x7f9aec003970 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): picked mon.b con 0x7f9aec0047c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): picked mon.c con 0x7f9aec005180 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040a7d70 con 0x7f9aec003970 2023-08-22T23:44:45.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00c020 con 0x7f9aec0047c0 2023-08-22T23:44:45.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00c1e0 con 0x7f9aec005180 2023-08-22T23:44:45.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b027fc640 10 monclient(hunting): _finish_auth 0 2023-08-22T23:44:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.709+0000 7f9b09b38640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): get_auth_request con 0x7f9aec005180 auth_method 0 2023-08-22T23:44:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-08-22T23:44:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.710+0000 7f9b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 10 monclient(hunting): handle_auth_done global_id 14775 payload 293 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 1 -- 172.21.15.29:0/1295064232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aec0047c0 msgr2=0x7f9aec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 1 -- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec003970 msgr2=0x7f9aec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec003970 0x7f9aec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 1 monclient: found mon.c 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 1 -- 172.21.15.29:0/1295064232 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec003520 con 0x7f9aec005180 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.712+0000 7f9b03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.713+0000 7f9b03fff640 1 -- 172.21.15.29:0/1295064232 reap_dead start 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.714+0000 7f9b03fff640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9aec00b980 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c240 tx=0x7f9aec0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382689ff73c25f40 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc005ad0 con 0x7f9aec005180 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: dump: 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc005c70 con 0x7f9aec005180 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 1 -- 172.21.15.29:0/1295064232 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc02c8b0 con 0x7f9aec005180 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 4 set_mon_vals no callback set 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b027fc640 10 monclient: dump: 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 10 monclient: shutdown 2023-08-22T23:44:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 1 -- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 msgr2=0x7f9aec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9aec00b980 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c240 tx=0x7f9aec0057f0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 1 -- 172.21.15.29:0/1295064232 shutdown_connections 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 1 --2- 172.21.15.29:0/1295064232 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 1 -- 172.21.15.29:0/1295064232 >> 172.21.15.29:0/1295064232 conn(0x7f9b04065fb0 msgr2=0x7f9b040b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.718+0000 7f9b09b38640 1 -- 172.21.15.29:0/1295064232 shutdown_connections 2023-08-22T23:44:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.719+0000 7f9b09b38640 1 -- 172.21.15.29:0/1295064232 wait complete. 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.719+0000 7f9b09b38640 10 monclient: build_initial_monmap 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.719+0000 7f9b09b38640 10 monclient: monmap: 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.719+0000 7f9b09b38640 1 librados: starting msgr at 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.719+0000 7f9b09b38640 1 librados: starting objecter 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 Processor -- start 2023-08-22T23:44:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 -- start start 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 librados: setting wanted keys 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 librados: calling monclient init 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient: init 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9b040acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): picked mon.noname-c con 0x7f9aec005180 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b041d9620 0x7f9b0413f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): picked mon.noname-b con 0x7f9b041d9620 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04141890 0x7f9b04141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): picked mon.noname-a con 0x7f9b04141890 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041400c0 con 0x7f9b04141890 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b04141140 con 0x7f9b041d9620 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0413a7b0 con 0x7f9aec005180 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.720+0000 7f9b09b38640 10 monclient(hunting): authenticate will time out at 2849.971298s 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9b040acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b03fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04141890 0x7f9b04141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b02ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b041d9620 0x7f9b0413f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9b040acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34278/0 (socket says 172.21.15.29:34278) 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 1 -- 172.21.15.29:0/3661779070 learned_addr learned my addr 172.21.15.29:0/3661779070 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): get_auth_request con 0x7f9aec005180 auth_method 0 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b03fff640 10 monclient(hunting): get_auth_request con 0x7f9b04141890 auth_method 0 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b03fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b03fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b02ffd640 10 monclient(hunting): get_auth_request con 0x7f9b041d9620 auth_method 0 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b02ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.721+0000 7f9b037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.722+0000 7f9b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.722+0000 7f9b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.722+0000 7f9b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.722+0000 7f9b02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.722+0000 7f9b02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.722+0000 7f9b02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 10 monclient(hunting): handle_auth_done global_id 16691 payload 824 2023-08-22T23:44:45.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 1 -- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b041d9620 msgr2=0x7f9b0413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b041d9620 0x7f9b0413f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 1 -- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04141890 msgr2=0x7f9b04141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b04141890 0x7f9b04141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 1 monclient: found mon.noname-c 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.724+0000 7f9b037fe640 1 -- 172.21.15.29:0/3661779070 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b04141370 con 0x7f9aec005180 2023-08-22T23:44:45.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.727+0000 7f9b037fe640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9b040acd10 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f9af400ef40 tx=0x7f9af400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.727+0000 7f9aebfff640 10 client.?.objecter ms_handle_connect 0x7f9aec005180 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.727+0000 7f9aebfff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4010040 con 0x7f9aec005180 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: dump: 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.744 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.745 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4014400 con 0x7f9aec005180 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4009c20 con 0x7f9aec005180 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 5 monclient: authenticate success, global_id 16691 2023-08-22T23:44:45.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 10 monclient: _renew_subs 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0413fac0 con 0x7f9aec005180 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b017fa640 4 set_mon_vals no callback set 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9aebfff640 10 monclient: dump: 2023-08-22T23:44:45.747 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.748 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 10 client.16691.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 10 monclient: _renew_subs 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b041d9c60 con 0x7f9aec005180 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.730+0000 7f9b09b38640 1 librados: init done 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.731+0000 7f9b09b38640 10 client.16691.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:45.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.731+0000 7f9b09b38640 15 client.16691.objecter _session_command_op_assign -1 1 2023-08-22T23:44:45.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.731+0000 7f9b09b38640 20 client.16691.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:45.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.731+0000 7f9b09b38640 10 client.16691.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:45.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.731+0000 7f9b09b38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.731+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b040a9df0 con 0x7f9aec005180 2023-08-22T23:44:45.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.751+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9af40147b0 con 0x7f9aec005180 2023-08-22T23:44:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.752+0000 7f9aebfff640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9af006ba30 0x7f9af006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.752+0000 7f9b02ffd640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9af006ba30 0x7f9af006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.752+0000 7f9b02ffd640 10 monclient: get_auth_request con 0x7f9af006ba30 auth_method 0 2023-08-22T23:44:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.753+0000 7f9b02ffd640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9af006ba30 0x7f9af006def0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9b041d9a00 tx=0x7f9af8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.753+0000 7f9aebfff640 10 client.16691.objecter ms_handle_connect 0x7f9af006ba30 2023-08-22T23:44:45.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f9af405c2e0 con 0x7f9aec005180 2023-08-22T23:44:45.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 10 client.16691.objecter ms_dispatch 0x7f9b04000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:45.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 3 client.16691.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:45.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 3 client.16691.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:45.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 10 client.16691.objecter checking command 1 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7f9af0073230 0x7f9af00756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 20 client.16691.objecter _get_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 15 client.16691.objecter _session_command_op_remove -1 1 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 20 client.16691.objecter _get_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 20 client.16691.objecter get_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 15 client.16691.objecter _session_command_op_assign 2 1 2023-08-22T23:44:45.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 4 2023-08-22T23:44:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 10 client.16691.objecter _send_command 1 2023-08-22T23:44:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 --> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9af0075da0 con 0x7f9af0073230 2023-08-22T23:44:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 20 client.16691.objecter dump_active .. 0 homeless 2023-08-22T23:44:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9af4054910 con 0x7f9aec005180 2023-08-22T23:44:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9aebfff640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9b03fff640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7f9af0073230 0x7f9af00756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.754+0000 7f9b03fff640 10 monclient: get_auth_request con 0x7f9af0073230 auth_method 0 2023-08-22T23:44:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.757+0000 7f9aebfff640 10 client.16691.objecter ms_handle_connect 0x7f9af0073230 2023-08-22T23:44:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.757+0000 7f9b03fff640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7f9af0073230 0x7f9af00756d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.783+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== osd.2 v2:172.21.15.29:6800/1029571786 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f9af0075da0 con 0x7f9af0073230 2023-08-22T23:44:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.783+0000 7f9aebfff640 10 client.16691.objecter ms_dispatch 0x7f9b04000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.783+0000 7f9aebfff640 10 client.16691.objecter _finish_command 1 = system:0 2023-08-22T23:44:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.783+0000 7f9aebfff640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 4 2023-08-22T23:44:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.783+0000 7f9aebfff640 15 client.16691.objecter _session_command_op_remove 2 1 2023-08-22T23:44:45.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 10 client.16691.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:45.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 15 client.16691.objecter _session_command_op_assign -1 2 2023-08-22T23:44:45.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 20 client.16691.objecter _get_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 20 client.16691.objecter _get_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 15 client.16691.objecter _session_command_op_remove -1 2 2023-08-22T23:44:45.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 20 client.16691.objecter get_session s=0x7f9af0072b60 osd=2 3 2023-08-22T23:44:45.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 15 client.16691.objecter _session_command_op_assign 2 2 2023-08-22T23:44:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 4 2023-08-22T23:44:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 10 client.16691.objecter _send_command 2 2023-08-22T23:44:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.795+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 --> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9b040a9bd0 con 0x7f9af0073230 2023-08-22T23:44:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9aebfff640 1 -- 172.21.15.29:0/3661779070 <== osd.2 v2:172.21.15.29:6800/1029571786 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9afc03c040 con 0x7f9af0073230 2023-08-22T23:44:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9aebfff640 10 client.16691.objecter ms_dispatch 0x7f9b04000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9aebfff640 10 client.16691.objecter _finish_command 2 = system:0 2023-08-22T23:44:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9aebfff640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 4 2023-08-22T23:44:45.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9aebfff640 15 client.16691.objecter _session_command_op_remove 2 2 2023-08-22T23:44:45.812 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:45.812 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:45.812 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 10 librados: watch_flush enter 2023-08-22T23:44:45.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 10 librados: watch_flush exit 2023-08-22T23:44:45.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 10 client.16691.objecter close_session for osd.2 2023-08-22T23:44:45.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7f9af0073230 msgr2=0x7f9af00756d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:6800/1029571786,v1:172.21.15.29:6801/1029571786] conn(0x7f9af0073230 0x7f9af00756d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 20 client.16691.objecter put_session s=0x7f9af0072b60 osd=2 1 2023-08-22T23:44:45.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 20 client.16691.objecter shutdown clearing up homeless session... 2023-08-22T23:44:45.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 10 client.16691.objecter successfully canceled tick 2023-08-22T23:44:45.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9af006ba30 msgr2=0x7f9af006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9af006ba30 0x7f9af006def0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9b041d9a00 tx=0x7f9af8009040 comp rx=0 tx=0).stop 2023-08-22T23:44:45.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 10 monclient: shutdown 2023-08-22T23:44:45.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:45.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 msgr2=0x7f9b040acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 --2- 172.21.15.29:0/3661779070 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9aec005180 0x7f9b040acd10 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f9af400ef40 tx=0x7f9af400b480 comp rx=0 tx=0).stop 2023-08-22T23:44:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b03fff640 1 -- 172.21.15.29:0/3661779070 reap_dead start 2023-08-22T23:44:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 shutdown_connections 2023-08-22T23:44:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 >> 172.21.15.29:0/3661779070 conn(0x7f9b04065fb0 msgr2=0x7f9b040b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 shutdown_connections 2023-08-22T23:44:45.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.810+0000 7f9b09b38640 1 -- 172.21.15.29:0/3661779070 wait complete. 2023-08-22T23:44:45.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.811+0000 7f9b09b38640 1 librados: shutdown 2023-08-22T23:44:45.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-08-22T23:44:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.933+0000 7f8c1a2a9640 10 monclient: get_monmap_and_config 2023-08-22T23:44:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.933+0000 7f8c1a2a9640 10 monclient: build_initial_monmap 2023-08-22T23:44:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.933+0000 7f8c1a2a9640 10 monclient: monmap: 2023-08-22T23:44:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:45.934893+0000 2023-08-22T23:44:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:45.934893+0000 2023-08-22T23:44:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:45.937 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 Processor -- start 2023-08-22T23:44:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 -- start start 2023-08-22T23:44:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient: init 2023-08-22T23:44:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f8c140adee0 0x7f8c140710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient(hunting): picked mon.noname-c con 0x7f8c140adee0 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 0x7f8c1413c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient(hunting): picked mon.noname-b con 0x7f8c140ae2c0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c1413c870 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c140a7f90 con 0x7f8c1413c870 2023-08-22T23:44:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1413eed0 con 0x7f8c140adee0 2023-08-22T23:44:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14142430 con 0x7f8c140ae2c0 2023-08-22T23:44:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.934+0000 7f8c1a2a9640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c1a2a9640 10 monclient(hunting): authenticate will time out at 2850.185476s 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 0x7f8c1413c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c13fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f8c140adee0 0x7f8c140710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 0x7f8c1413c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34290/0 (socket says 172.21.15.29:34290) 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 1 -- 172.21.15.29:0/3573242795 learned_addr learned my addr 172.21.15.29:0/3573242795 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 10 monclient(hunting): get_auth_request con 0x7f8c140ae2c0 auth_method 0 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.935+0000 7f8c137fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c13fff640 10 monclient(hunting): get_auth_request con 0x7f8c140adee0 auth_method 0 2023-08-22T23:44:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c13fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.936+0000 7f8c13fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.937+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.937+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.937+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 10 monclient(hunting): handle_auth_done global_id 16694 payload 274 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 1 -- 172.21.15.29:0/3573242795 >> v2:172.21.15.52:3300/0 conn(0x7f8c140adee0 msgr2=0x7f8c140710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 1 --2- 172.21.15.29:0/3573242795 >> v2:172.21.15.52:3300/0 conn(0x7f8c140adee0 0x7f8c140710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 1 -- 172.21.15.29:0/3573242795 >> v1:172.21.15.52:6789/0 conn(0x7f8c1413c870 legacy=0x7f8c1413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 1 monclient: found mon.noname-b 2023-08-22T23:44:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.938+0000 7f8c137fe640 1 -- 172.21.15.29:0/3573242795 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c14142570 con 0x7f8c140ae2c0 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.940+0000 7f8c137fe640 1 --2- 172.21.15.29:0/3573242795 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 0x7f8c1413c330 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009870 tx=0x7f8c0402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=966422dab6ca8a1d server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 1 -- 172.21.15.29:0/3573242795 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040317c0 con 0x7f8c140ae2c0 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: dump: 2023-08-22T23:44:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 1 -- 172.21.15.29:0/3573242795 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04031920 con 0x7f8c140ae2c0 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 1 -- 172.21.15.29:0/3573242795 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04031cb0 con 0x7f8c140ae2c0 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c12ffd640 10 monclient: dump: 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c1a2a9640 5 monclient: authenticate success, global_id 16694 2023-08-22T23:44:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c1a2a9640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.943+0000 7f8c1a2a9640 4 set_mon_vals no callback set 2023-08-22T23:44:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 10 monclient: shutdown 2023-08-22T23:44:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/3573242795 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 msgr2=0x7f8c1413c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/3573242795 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 0x7f8c1413c330 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009870 tx=0x7f8c0402b860 comp rx=0 tx=0).stop 2023-08-22T23:44:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/3573242795 shutdown_connections 2023-08-22T23:44:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/3573242795 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c140ae2c0 0x7f8c1413c330 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/3573242795 >> v2:172.21.15.52:3300/0 conn(0x7f8c140adee0 0x7f8c140710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/3573242795 >> 172.21.15.29:0/3573242795 conn(0x7f8c140afad0 msgr2=0x7f8c140b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/3573242795 shutdown_connections 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.944+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/3573242795 wait complete. 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 10 monclient: build_initial_monmap 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 10 monclient: monmap: 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 1 librados: starting msgr at 2023-08-22T23:44:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 1 librados: starting objecter 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 1 Processor -- start 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 1 -- start start 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 1 librados: setting wanted keys 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 1 librados: calling monclient init 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.945+0000 7f8c1a2a9640 10 monclient: init 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:44:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c140adee0 0x7f8c1413e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): picked mon.noname-b con 0x7f8c140adee0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1413c870 0x7f8c141d3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): picked mon.noname-a con 0x7f8c1413c870 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 0x7f8c141d4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): picked mon.noname-c con 0x7f8c14141680 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141d17f0 con 0x7f8c1413c870 2023-08-22T23:44:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1413e780 con 0x7f8c140adee0 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140a8750 con 0x7f8c14141680 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c1a2a9640 10 monclient(hunting): authenticate will time out at 2850.196953s 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c18846640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 0x7f8c141d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.946+0000 7f8c137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1413c870 0x7f8c141d3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c13fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c140adee0 0x7f8c1413e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 0x7f8c141d4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34306/0 (socket says 172.21.15.29:34306) 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 1 -- 172.21.15.29:0/2427172129 learned_addr learned my addr 172.21.15.29:0/2427172129 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): get_auth_request con 0x7f8c14141680 auth_method 0 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): get_auth_request con 0x7f8c1413c870 auth_method 0 2023-08-22T23:44:45.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c13fff640 10 monclient(hunting): get_auth_request con 0x7f8c140adee0 auth_method 0 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c13fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c13fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c18846640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.947+0000 7f8c137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.948+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.948+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:45.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.948+0000 7f8c13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 10 monclient(hunting): handle_auth_done global_id 14781 payload 824 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 1 -- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 msgr2=0x7f8c141d4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 0x7f8c141d4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 1 -- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c140adee0 msgr2=0x7f8c1413e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c140adee0 0x7f8c1413e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 1 monclient: found mon.noname-a 2023-08-22T23:44:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:45.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c137fe640 1 -- 172.21.15.29:0/2427172129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040094d0 con 0x7f8c1413c870 2023-08-22T23:44:45.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.950+0000 7f8c18846640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 0x7f8c141d4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:45.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.952+0000 7f8c137fe640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1413c870 0x7f8c141d3b50 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f8c04033000 tx=0x7f8c040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.952+0000 7f8c10ff9640 10 client.?.objecter ms_handle_connect 0x7f8c1413c870 2023-08-22T23:44:45.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.952+0000 7f8c10ff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:45.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04032860 con 0x7f8c1413c870 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: dump: 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.967 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: _finish_auth 0 2023-08-22T23:44:45.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: _check_auth_tickets 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040329e0 con 0x7f8c1413c870 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04032d90 con 0x7f8c1413c870 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 5 monclient: authenticate success, global_id 14781 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 10 monclient: _renew_subs 2023-08-22T23:44:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c140a8420 con 0x7f8c1413c870 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c127fc640 4 set_mon_vals no callback set 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c10ff9640 10 monclient: dump: 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:45.971 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 10 client.14781.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 10 monclient: _renew_subs 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c140a8c70 con 0x7f8c1413c870 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.956+0000 7f8c1a2a9640 1 librados: init done 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.957+0000 7f8c1a2a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.957+0000 7f8c1a2a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.957+0000 7f8c1a2a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.957+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c140a85b0 con 0x7f8c1413c870 2023-08-22T23:44:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.975+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8c0404e450 con 0x7f8c1413c870 2023-08-22T23:44:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.976+0000 7f8c10ff9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8bf406b9e0 0x7f8bf406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.976+0000 7f8c13fff640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8bf406b9e0 0x7f8bf406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.976+0000 7f8c13fff640 10 monclient: get_auth_request con 0x7f8bf406b9e0 auth_method 0 2023-08-22T23:44:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.977+0000 7f8c10ff9640 10 client.14781.objecter ms_handle_connect 0x7f8bf406b9e0 2023-08-22T23:44:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.977+0000 7f8c13fff640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8bf406b9e0 0x7f8bf406dea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009b40 tx=0x7f8bfc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.978+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f8c0407d8c0 con 0x7f8c1413c870 2023-08-22T23:44:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.978+0000 7f8c10ff9640 10 client.14781.objecter ms_dispatch 0x7f8c14000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.978+0000 7f8c10ff9640 3 client.14781.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.978+0000 7f8c10ff9640 3 client.14781.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:45.978+0000 7f8c10ff9640 20 client.14781.objecter dump_active .. 0 homeless 2023-08-22T23:44:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.242+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f8c04075ef0 con 0x7f8c1413c870 2023-08-22T23:44:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.242+0000 7f8c10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.242+0000 7f8c10ff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:46.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.427+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c14071790 con 0x7f8bf406b9e0 2023-08-22T23:44:46.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.428+0000 7f8c10ff9640 1 -- 172.21.15.29:0/2427172129 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8bfc011a60 con 0x7f8bf406b9e0 2023-08-22T23:44:46.429 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 2 to deep-scrub 2023-08-22T23:44:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.429+0000 7f8c1a2a9640 10 librados: watch_flush enter 2023-08-22T23:44:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.429+0000 7f8c1a2a9640 10 librados: watch_flush exit 2023-08-22T23:44:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.429+0000 7f8c1a2a9640 20 client.14781.objecter shutdown clearing up homeless session... 2023-08-22T23:44:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.429+0000 7f8c1a2a9640 10 client.14781.objecter successfully canceled tick 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8bf406b9e0 msgr2=0x7f8bf406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8bf406b9e0 0x7f8bf406dea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009b40 tx=0x7f8bfc009190 comp rx=0 tx=0).stop 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 10 monclient: shutdown 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1413c870 msgr2=0x7f8c141d3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1413c870 0x7f8c141d3b50 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f8c04033000 tx=0x7f8c040046c0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 shutdown_connections 2023-08-22T23:44:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f8bf406b9e0 0x7f8bf406dea0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f8c14141680 0x7f8c141d4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1413c870 0x7f8c141d3b50 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 --2- 172.21.15.29:0/2427172129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c140adee0 0x7f8c1413e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 >> 172.21.15.29:0/2427172129 conn(0x7f8c140afad0 msgr2=0x7f8c140707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 shutdown_connections 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 -- 172.21.15.29:0/2427172129 wait complete. 2023-08-22T23:44:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.430+0000 7f8c1a2a9640 1 librados: shutdown 2023-08-22T23:44:46.443 INFO:tasks.ceph:Scrubbing osd.3 2023-08-22T23:44:46.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.555+0000 7fe40452d640 10 monclient: get_monmap_and_config 2023-08-22T23:44:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.555+0000 7fe40452d640 10 monclient: build_initial_monmap 2023-08-22T23:44:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.555+0000 7fe40452d640 10 monclient: monmap: 2023-08-22T23:44:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:46.556602+0000 2023-08-22T23:44:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:46.556602+0000 2023-08-22T23:44:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:46.558 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.555+0000 7fe40452d640 1 Processor -- start 2023-08-22T23:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.555+0000 7fe40452d640 1 -- start start 2023-08-22T23:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.555+0000 7fe40452d640 10 monclient: init 2023-08-22T23:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-08-22T23:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3fc0a7b90 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3fc0a7f70 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fe3fc13e800 0x7fe3fc14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc13e800 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc0b0d50 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc0a9820 con 0x7fe3fc0a7b90 2023-08-22T23:44:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3fc0a9b00 con 0x7fe3fc13e800 2023-08-22T23:44:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.556+0000 7fe40452d640 10 monclient(hunting): authenticate will time out at 2850.807183s 2023-08-22T23:44:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.557+0000 7fe401ac8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe3fc0a7f70 0x7fe3fc13e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48112/0 (socket says 172.21.15.29:48112) 2023-08-22T23:44:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.557+0000 7fe401ac8640 1 -- 172.21.15.29:0/1387749713 learned_addr learned my addr 172.21.15.29:0/1387749713 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:46.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.557+0000 7fe402aca640 1 --2- 172.21.15.29:0/1387749713 >> v2:172.21.15.52:3300/0 conn(0x7fe3fc13e800 0x7fe3fc14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.558+0000 7fe402aca640 10 monclient(hunting): get_auth_request con 0x7fe3fc13e800 auth_method 0 2023-08-22T23:44:46.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.558+0000 7fe402aca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.558+0000 7fe402aca640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.558+0000 7fe402aca640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe402aca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe402aca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe402aca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe4012c7640 1 -- 172.21.15.29:0/1387749713 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430118347 0 0) 0x7fe3fc0b0d50 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe4012c7640 10 monclient(hunting): _negotiate 2023-08-22T23:44:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe4012c7640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe4012c7640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe4012c7640 10 monclient(hunting): my global_id is 14784 2023-08-22T23:44:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.559+0000 7fe4012c7640 1 -- 172.21.15.29:0/1387749713 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e8003520 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 10 monclient(hunting): handle_auth_done global_id 19600 payload 274 2023-08-22T23:44:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 1 -- 172.21.15.29:0/1387749713 >> v1:172.21.15.52:6789/0 conn(0x7fe3fc0a7b90 legacy=0x7fe3fc141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 1 -- 172.21.15.29:0/1387749713 >> v1:172.21.15.29:6789/0 conn(0x7fe3fc0a7f70 legacy=0x7fe3fc13e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 10 monclient: _finish_hunting 0 2023-08-22T23:44:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 1 monclient: found mon.noname-c 2023-08-22T23:44:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.562+0000 7fe402aca640 1 -- 172.21.15.29:0/1387749713 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc0a9de0 con 0x7fe3fc13e800 2023-08-22T23:44:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.566+0000 7fe402aca640 1 --2- 172.21.15.29:0/1387749713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc13e800 0x7fe3fc14ca80 secure :-1 s=READY pgs=4629 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009d90 tx=0x7fe3f402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fad1e412e86579e8 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 1 -- 172.21.15.29:0/1387749713 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f402cc60 con 0x7fe3fc13e800 2023-08-22T23:44:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: dump: 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: _finish_auth 0 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: _check_auth_tickets 2023-08-22T23:44:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 1 -- 172.21.15.29:0/1387749713 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f402cdc0 con 0x7fe3fc13e800 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 1 -- 172.21.15.29:0/1387749713 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4034660 con 0x7fe3fc13e800 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 5 monclient: authenticate success, global_id 19600 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 4 set_mon_vals no callback set 2023-08-22T23:44:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe4012c7640 10 monclient: dump: 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.577 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 10 monclient: shutdown 2023-08-22T23:44:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 1 -- 172.21.15.29:0/1387749713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc13e800 msgr2=0x7fe3fc14ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.571+0000 7fe40452d640 1 --2- 172.21.15.29:0/1387749713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc13e800 0x7fe3fc14ca80 secure :-1 s=READY pgs=4629 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009d90 tx=0x7fe3f402beb0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 1 -- 172.21.15.29:0/1387749713 shutdown_connections 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 1 --2- 172.21.15.29:0/1387749713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc13e800 0x7fe3fc14ca80 unknown :-1 s=CLOSED pgs=4629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 1 -- 172.21.15.29:0/1387749713 >> 172.21.15.29:0/1387749713 conn(0x7fe3fc0b6920 msgr2=0x7fe3fc0b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 1 -- 172.21.15.29:0/1387749713 shutdown_connections 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 1 -- 172.21.15.29:0/1387749713 wait complete. 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 10 monclient: build_initial_monmap 2023-08-22T23:44:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.572+0000 7fe40452d640 10 monclient: monmap: 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 librados: starting msgr at 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 librados: starting objecter 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 Processor -- start 2023-08-22T23:44:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 -- start start 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 librados: setting wanted keys 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 librados: calling monclient init 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient: init 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc0a7b90 0x7fe3fc13bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3fc0a7b90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc0a7f70 0x7fe3fc1da150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3fc0a7f70 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe3fc13e800 0x7fe3fc137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc13e800 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc140ec0 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc1d99e0 con 0x7fe3fc0a7b90 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc1d9cc0 con 0x7fe3fc13e800 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.573+0000 7fe40452d640 10 monclient(hunting): authenticate will time out at 2850.824350s 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc0a7f70 0x7fe3fc1da150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe402aca640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe3fc13e800 0x7fe3fc137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe4022c9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc0a7b90 0x7fe3fc13bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc0a7f70 0x7fe3fc1da150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52690/0 (socket says 172.21.15.29:52690) 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 1 -- 172.21.15.29:0/1467483078 learned_addr learned my addr 172.21.15.29:0/1467483078 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a7f70 auth_method 0 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe401ac8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe402aca640 10 monclient(hunting): get_auth_request con 0x7fe3fc13e800 auth_method 0 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe402aca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe402aca640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.574+0000 7fe402aca640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe4022c9640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a7b90 auth_method 0 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe4022c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe4022c9640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe4022c9640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe401ac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe401ac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe401ac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe402aca640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe402aca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.575+0000 7fe402aca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.576+0000 7fe4022c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.576+0000 7fe4022c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.576+0000 7fe4022c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 10 monclient(hunting): handle_auth_done global_id 14787 payload 824 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 1 -- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe3fc13e800 msgr2=0x7fe3fc137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe3fc13e800 0x7fe3fc137ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 1 -- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc0a7b90 msgr2=0x7fe3fc13bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3fc0a7b90 0x7fe3fc13bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 10 monclient: _finish_hunting 0 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 1 monclient: found mon.noname-a 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe401ac8640 1 -- 172.21.15.29:0/1467483078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4009a70 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.578+0000 7fe402aca640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe3fc13e800 0x7fe3fc137ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.580+0000 7fe401ac8640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc0a7f70 0x7fe3fc1da150 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00c530 tx=0x7fe3ec00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.580+0000 7fe3f2ffd640 10 client.?.objecter ms_handle_connect 0x7fe3fc0a7f70 2023-08-22T23:44:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.580+0000 7fe3f2ffd640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec005ce0 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: dump: 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec005e40 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 5 monclient: authenticate success, global_id 14787 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 10 monclient: _renew_subs 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3fc13d5f0 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec0072b0 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe3f2ffd640 10 monclient: dump: 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe400ac6640 4 set_mon_vals no callback set 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 10 client.14787.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 10 monclient: _renew_subs 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3fc138010 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.586+0000 7fe40452d640 1 librados: init done 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.587+0000 7fe40452d640 10 client.14787.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:46.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.587+0000 7fe40452d640 15 client.14787.objecter _session_command_op_assign -1 1 2023-08-22T23:44:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.587+0000 7fe40452d640 20 client.14787.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.587+0000 7fe40452d640 10 client.14787.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.587+0000 7fe40452d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.587+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe3fc0aa0c0 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.624+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe3ec007410 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.624+0000 7fe3f2ffd640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe3e006ba30 0x7fe3e006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.625+0000 7fe4022c9640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe3e006ba30 0x7fe3e006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.625+0000 7fe4022c9640 10 monclient: get_auth_request con 0x7fe3e006ba30 auth_method 0 2023-08-22T23:44:46.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.625+0000 7fe4022c9640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe3e006ba30 0x7fe3e006def0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000f60 tx=0x7fe3f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.625+0000 7fe3f2ffd640 10 client.14787.objecter ms_handle_connect 0x7fe3e006ba30 2023-08-22T23:44:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fe3ec05b440 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 10 client.14787.objecter ms_dispatch 0x7fe3fc000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 3 client.14787.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 3 client.14787.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 10 client.14787.objecter checking command 1 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7fe3e0073230 0x7fe3e00756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 20 client.14787.objecter _get_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 15 client.14787.objecter _session_command_op_remove -1 1 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 20 client.14787.objecter _get_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 20 client.14787.objecter get_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 15 client.14787.objecter _session_command_op_assign 3 1 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 4 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 10 client.14787.objecter _send_command 1 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 --> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3e0075da0 con 0x7fe3e0073230 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 20 client.14787.objecter dump_active .. 0 homeless 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3ec057500 con 0x7fe3fc0a7f70 2023-08-22T23:44:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe3f2ffd640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.628+0000 7fe402aca640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7fe3e0073230 0x7fe3e00756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.629+0000 7fe402aca640 10 monclient: get_auth_request con 0x7fe3e0073230 auth_method 0 2023-08-22T23:44:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.631+0000 7fe402aca640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7fe3e0073230 0x7fe3e00756d0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.631+0000 7fe3f2ffd640 10 client.14787.objecter ms_handle_connect 0x7fe3e0073230 2023-08-22T23:44:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.659+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== osd.3 v2:172.21.15.29:6815/4227673673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fe3e0075da0 con 0x7fe3e0073230 2023-08-22T23:44:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.659+0000 7fe3f2ffd640 10 client.14787.objecter ms_dispatch 0x7fe3fc000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.659+0000 7fe3f2ffd640 10 client.14787.objecter _finish_command 1 = system:0 2023-08-22T23:44:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.659+0000 7fe3f2ffd640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 4 2023-08-22T23:44:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.659+0000 7fe3f2ffd640 15 client.14787.objecter _session_command_op_remove 3 1 2023-08-22T23:44:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 10 client.14787.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 15 client.14787.objecter _session_command_op_assign -1 2 2023-08-22T23:44:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 20 client.14787.objecter _get_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 20 client.14787.objecter _get_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 15 client.14787.objecter _session_command_op_remove -1 2 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 20 client.14787.objecter get_session s=0x7fe3e0072b60 osd=3 3 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 15 client.14787.objecter _session_command_op_assign 3 2 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 4 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 10 client.14787.objecter _send_command 2 2023-08-22T23:44:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.672+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 --> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe3fc04dbd0 con 0x7fe3e0073230 2023-08-22T23:44:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.686+0000 7fe3f2ffd640 1 -- 172.21.15.29:0/1467483078 <== osd.3 v2:172.21.15.29:6815/4227673673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe3fc04dbd0 con 0x7fe3e0073230 2023-08-22T23:44:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.686+0000 7fe3f2ffd640 10 client.14787.objecter ms_dispatch 0x7fe3fc000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.686+0000 7fe3f2ffd640 10 client.14787.objecter _finish_command 2 = system:0 2023-08-22T23:44:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.686+0000 7fe3f2ffd640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 4 2023-08-22T23:44:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.686+0000 7fe3f2ffd640 15 client.14787.objecter _session_command_op_remove 3 2 2023-08-22T23:44:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:46.689 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:46.689 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 10 librados: watch_flush enter 2023-08-22T23:44:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 10 librados: watch_flush exit 2023-08-22T23:44:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 10 client.14787.objecter close_session for osd.3 2023-08-22T23:44:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7fe3e0073230 msgr2=0x7fe3e00756d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:6815/4227673673,v1:172.21.15.29:6817/4227673673] conn(0x7fe3e0073230 0x7fe3e00756d0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 20 client.14787.objecter put_session s=0x7fe3e0072b60 osd=3 1 2023-08-22T23:44:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 20 client.14787.objecter shutdown clearing up homeless session... 2023-08-22T23:44:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 10 client.14787.objecter successfully canceled tick 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe3e006ba30 msgr2=0x7fe3e006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe3e006ba30 0x7fe3e006def0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000f60 tx=0x7fe3f8009040 comp rx=0 tx=0).stop 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 10 monclient: shutdown 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc0a7f70 msgr2=0x7fe3fc1da150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 --2- 172.21.15.29:0/1467483078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc0a7f70 0x7fe3fc1da150 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00c530 tx=0x7fe3ec00c9f0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe402aca640 1 -- 172.21.15.29:0/1467483078 reap_dead start 2023-08-22T23:44:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 shutdown_connections 2023-08-22T23:44:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 >> 172.21.15.29:0/1467483078 conn(0x7fe3fc0b6920 msgr2=0x7fe3fc1cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 shutdown_connections 2023-08-22T23:44:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 -- 172.21.15.29:0/1467483078 wait complete. 2023-08-22T23:44:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.687+0000 7fe40452d640 1 librados: shutdown 2023-08-22T23:44:46.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-08-22T23:44:46.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.820+0000 7f6ff6d42640 10 monclient: get_monmap_and_config 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.820+0000 7f6ff6d42640 10 monclient: build_initial_monmap 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.820+0000 7f6ff6d42640 10 monclient: monmap: 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:46.822036+0000 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:46.822036+0000 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:46.822 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:46.823 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.821+0000 7f6ff6d42640 1 Processor -- start 2023-08-22T23:44:46.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.821+0000 7f6ff6d42640 1 -- start start 2023-08-22T23:44:46.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.821+0000 7f6ff6d42640 10 monclient: init 2023-08-22T23:44:46.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:46.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-08-22T23:44:46.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ff00a7d50 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ff00a8130 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6ff01418f0 0x7f6ff014ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff01418f0 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff013ee50 con 0x7f6ff00a8130 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00a8580 con 0x7f6ff00a7d50 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff00a87c0 con 0x7f6ff01418f0 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff6d42640 10 monclient(hunting): authenticate will time out at 2851.072786s 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.822+0000 7f6ff52df640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6ff01418f0 0x7f6ff014ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.823+0000 7f6ff4ade640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6ff00a7d50 0x7f6ff00aff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:43812/0 (socket says 172.21.15.29:43812) 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.823+0000 7f6ff4ade640 1 -- 172.21.15.29:0/3898001063 learned_addr learned my addr 172.21.15.29:0/3898001063 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.823+0000 7f6ff52df640 10 monclient(hunting): get_auth_request con 0x7f6ff01418f0 auth_method 0 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.823+0000 7f6ff52df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.823+0000 7f6ff52df640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.823+0000 7f6ff52df640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.824+0000 7f6ff52df640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.824+0000 7f6ff52df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.824+0000 7f6ff52df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 10 monclient(hunting): handle_auth_done global_id 19609 payload 274 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 1 -- 172.21.15.29:0/3898001063 >> v1:172.21.15.52:6789/0 conn(0x7f6ff00a7d50 legacy=0x7f6ff00aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 1 -- 172.21.15.29:0/3898001063 >> v1:172.21.15.29:6789/0 conn(0x7f6ff00a8130 legacy=0x7f6ff01411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 10 monclient: _finish_hunting 0 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 1 monclient: found mon.noname-c 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:46.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:46.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.827+0000 7f6ff52df640 1 -- 172.21.15.29:0/3898001063 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff00a9df0 con 0x7f6ff01418f0 2023-08-22T23:44:46.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.830+0000 7f6ff52df640 1 --2- 172.21.15.29:0/3898001063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff014ca20 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009d90 tx=0x7f6fe802bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90409b2ec2184819 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 1 -- 172.21.15.29:0/3898001063 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe802cb40 con 0x7f6ff01418f0 2023-08-22T23:44:46.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: dump: 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.836 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: _finish_auth 0 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:44:46.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6ff6d42640 5 monclient: authenticate success, global_id 19609 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 1 -- 172.21.15.29:0/3898001063 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe802cca0 con 0x7f6ff01418f0 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6ff6d42640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6ff6d42640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6ff6d42640 4 set_mon_vals no callback set 2023-08-22T23:44:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 1 -- 172.21.15.29:0/3898001063 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8034660 con 0x7f6ff01418f0 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.834+0000 7f6fef7fe640 10 monclient: dump: 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.840 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.840 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 10 monclient: shutdown 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 -- 172.21.15.29:0/3898001063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 msgr2=0x7f6ff014ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/3898001063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff014ca20 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009d90 tx=0x7f6fe802bd90 comp rx=0 tx=0).stop 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 -- 172.21.15.29:0/3898001063 shutdown_connections 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/3898001063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff014ca20 unknown :-1 s=CLOSED pgs=4631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 -- 172.21.15.29:0/3898001063 >> 172.21.15.29:0/3898001063 conn(0x7f6ff0065dd0 msgr2=0x7f6ff00b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 -- 172.21.15.29:0/3898001063 shutdown_connections 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.835+0000 7f6ff6d42640 1 -- 172.21.15.29:0/3898001063 wait complete. 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 10 monclient: build_initial_monmap 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 10 monclient: monmap: 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 1 librados: starting msgr at 2023-08-22T23:44:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 1 librados: starting objecter 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 1 Processor -- start 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 1 -- start start 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 1 librados: setting wanted keys 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 1 librados: calling monclient init 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.836+0000 7f6ff6d42640 10 monclient: init 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:44:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00a7d50 0x7f6ff00ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff00a7d50 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff00a8130 0x7f6ff01c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff00a8130 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff01c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff01418f0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00b06a0 con 0x7f6ff00a7d50 2023-08-22T23:44:46.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff01c9100 con 0x7f6ff01418f0 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00acd80 con 0x7f6ff00a8130 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.837+0000 7f6ff6d42640 10 monclient(hunting): authenticate will time out at 2851.087884s 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00a7d50 0x7f6ff00ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00a7d50 0x7f6ff00ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52692/0 (socket says 172.21.15.29:52692) 2023-08-22T23:44:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 1 -- 172.21.15.29:0/123336753 learned_addr learned my addr 172.21.15.29:0/123336753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 10 monclient(hunting): get_auth_request con 0x7f6ff00a7d50 auth_method 0 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.838+0000 7f6ff4ade640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6feffff640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff00a8130 0x7f6ff01c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6feffff640 10 monclient(hunting): get_auth_request con 0x7f6ff00a8130 auth_method 0 2023-08-22T23:44:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6feffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6feffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6ff4ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6ff4ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.839+0000 7f6ff4ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.840+0000 7f6feffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.840+0000 7f6feffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.840+0000 7f6feffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.841+0000 7f6ff52df640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff01c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 10 monclient(hunting): handle_auth_done global_id 16703 payload 824 2023-08-22T23:44:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 1 -- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 msgr2=0x7f6ff01c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff01c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 1 -- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00a7d50 msgr2=0x7f6ff00ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00a7d50 0x7f6ff00ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 1 monclient: found mon.noname-c 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.843+0000 7f6feffff640 1 -- 172.21.15.29:0/123336753 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009a70 con 0x7f6ff00a8130 2023-08-22T23:44:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.845+0000 7f6feffff640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff00a8130 0x7f6ff01c72f0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c410 tx=0x7f6fdc00c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.845+0000 7f6fed7fa640 10 client.?.objecter ms_handle_connect 0x7f6ff00a8130 2023-08-22T23:44:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.845+0000 7f6fed7fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0035f0 con 0x7f6ff00a8130 2023-08-22T23:44:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: dump: 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: _finish_auth 0 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: _check_auth_tickets 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 5 monclient: authenticate success, global_id 16703 2023-08-22T23:44:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 10 monclient: _renew_subs 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff01c8d30 con 0x7f6ff00a8130 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 10 client.16703.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 10 monclient: _renew_subs 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff01c9870 con 0x7f6ff00a8130 2023-08-22T23:44:46.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6ff6d42640 1 librados: init done 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc003750 con 0x7f6ff00a8130 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc003b40 con 0x7f6ff00a8130 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6fed7fa640 10 monclient: dump: 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:46.858 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:46.859 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:46.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.849+0000 7f6feeffd640 4 set_mon_vals no callback set 2023-08-22T23:44:46.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.850+0000 7f6ff6d42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:46.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.850+0000 7f6ff6d42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:46.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.850+0000 7f6ff6d42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:46.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.850+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0005040 con 0x7f6ff00a8130 2023-08-22T23:44:46.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.870+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6fdc003ca0 con 0x7f6ff00a8130 2023-08-22T23:44:46.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.871+0000 7f6fed7fa640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6fd406b9e0 0x7f6fd406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:46.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.871+0000 7f6ff4ade640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6fd406b9e0 0x7f6fd406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:46.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.871+0000 7f6ff4ade640 10 monclient: get_auth_request con 0x7f6fd406b9e0 auth_method 0 2023-08-22T23:44:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6ff4ade640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6fd406b9e0 0x7f6fd406dea0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4005880 tx=0x7f6fe4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6fed7fa640 10 client.16703.objecter ms_handle_connect 0x7f6fd406b9e0 2023-08-22T23:44:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f6fdc05aee0 con 0x7f6ff00a8130 2023-08-22T23:44:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6fed7fa640 10 client.16703.objecter ms_dispatch 0x7f6ff0000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6fed7fa640 3 client.16703.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6fed7fa640 3 client.16703.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:46.872+0000 7f6fed7fa640 20 client.16703.objecter dump_active .. 0 homeless 2023-08-22T23:44:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.121+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f6fdc053510 con 0x7f6ff00a8130 2023-08-22T23:44:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.121+0000 7f6fed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.121+0000 7f6fed7fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:47.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.301+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fd0000bc0 con 0x7f6fd406b9e0 2023-08-22T23:44:47.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.301+0000 7f6fed7fa640 1 -- 172.21.15.29:0/123336753 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6fd0000bc0 con 0x7f6fd406b9e0 2023-08-22T23:44:47.303 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 3 to deep-scrub 2023-08-22T23:44:47.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 10 librados: watch_flush enter 2023-08-22T23:44:47.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 10 librados: watch_flush exit 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 20 client.16703.objecter shutdown clearing up homeless session... 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 10 client.16703.objecter successfully canceled tick 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6fd406b9e0 msgr2=0x7f6fd406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6fd406b9e0 0x7f6fd406dea0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4005880 tx=0x7f6fe4009040 comp rx=0 tx=0).stop 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 10 monclient: shutdown 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:47.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff00a8130 msgr2=0x7f6ff01c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff00a8130 0x7f6ff01c72f0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c410 tx=0x7f6fdc00c810 comp rx=0 tx=0).stop 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 shutdown_connections 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f6fd406b9e0 0x7f6fd406dea0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ff01418f0 0x7f6ff01c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f6ff00a8130 0x7f6ff01c72f0 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 --2- 172.21.15.29:0/123336753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00a7d50 0x7f6ff00ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 >> 172.21.15.29:0/123336753 conn(0x7f6ff0065dd0 msgr2=0x7f6ff01cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:47.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 shutdown_connections 2023-08-22T23:44:47.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 -- 172.21.15.29:0/123336753 wait complete. 2023-08-22T23:44:47.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.304+0000 7f6ff6d42640 1 librados: shutdown 2023-08-22T23:44:47.316 INFO:tasks.ceph:Scrubbing osd.4 2023-08-22T23:44:47.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.429+0000 7f1e873e8640 10 monclient: get_monmap_and_config 2023-08-22T23:44:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.429+0000 7f1e873e8640 10 monclient: build_initial_monmap 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.429+0000 7f1e873e8640 10 monclient: monmap: 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:47.430789+0000 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:47.430789+0000 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:47.432 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.430+0000 7f1e873e8640 1 Processor -- start 2023-08-22T23:44:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.430+0000 7f1e873e8640 1 -- start start 2023-08-22T23:44:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.430+0000 7f1e873e8640 10 monclient: init 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e800a81b0 0x7f1e800a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): picked mon.noname-b con 0x7f1e800a81b0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e8013e760 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1e8013eba0 0x7f1e80144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1e8013eba0 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e800b07e0 con 0x7f1e8013e760 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e800a7d70 con 0x7f1e8013eba0 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800a98a0 con 0x7f1e800a81b0 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.431+0000 7f1e873e8640 10 monclient(hunting): authenticate will time out at 2851.681863s 2023-08-22T23:44:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e84983640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1e8013e760 0x7f1e80144720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48128/0 (socket says 172.21.15.29:48128) 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e84983640 1 -- 172.21.15.29:0/460693982 learned_addr learned my addr 172.21.15.29:0/460693982 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e85985640 1 --2- 172.21.15.29:0/460693982 >> v2:172.21.15.52:3300/0 conn(0x7f1e8013eba0 0x7f1e80144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e85184640 1 --2- 172.21.15.29:0/460693982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e800a81b0 0x7f1e800a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e85184640 10 monclient(hunting): get_auth_request con 0x7f1e800a81b0 auth_method 0 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e85184640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e85184640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.432+0000 7f1e85184640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85985640 10 monclient(hunting): get_auth_request con 0x7f1e8013eba0 auth_method 0 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85985640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85985640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85985640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85184640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85184640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85184640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.433+0000 7f1e85985640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.434+0000 7f1e85985640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.434+0000 7f1e85985640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 10 monclient(hunting): handle_auth_done global_id 16706 payload 274 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 1 -- 172.21.15.29:0/460693982 >> v2:172.21.15.52:3300/0 conn(0x7f1e8013eba0 msgr2=0x7f1e80144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 1 --2- 172.21.15.29:0/460693982 >> v2:172.21.15.52:3300/0 conn(0x7f1e8013eba0 0x7f1e80144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 1 -- 172.21.15.29:0/460693982 >> v1:172.21.15.29:6789/0 conn(0x7f1e8013e760 legacy=0x7f1e80144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 10 monclient: _finish_hunting 0 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 1 monclient: found mon.noname-b 2023-08-22T23:44:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.435+0000 7f1e85184640 1 -- 172.21.15.29:0/460693982 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e800a8ad0 con 0x7f1e800a81b0 2023-08-22T23:44:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.437+0000 7f1e85184640 1 --2- 172.21.15.29:0/460693982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e800a81b0 0x7f1e800a8590 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0097a0 tx=0x7f1e7c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2858bb3511ab53b server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 1 -- 172.21.15.29:0/460693982 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c02d040 con 0x7f1e800a81b0 2023-08-22T23:44:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: dump: 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: _finish_auth 0 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 1 -- 172.21.15.29:0/460693982 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c031810 con 0x7f1e800a81b0 2023-08-22T23:44:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e873e8640 5 monclient: authenticate success, global_id 16706 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e873e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e873e8640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 1 -- 172.21.15.29:0/460693982 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c031ba0 con 0x7f1e800a81b0 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e873e8640 4 set_mon_vals no callback set 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.441+0000 7f1e77fff640 10 monclient: dump: 2023-08-22T23:44:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.447 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 10 monclient: shutdown 2023-08-22T23:44:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 -- 172.21.15.29:0/460693982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e800a81b0 msgr2=0x7f1e800a8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 --2- 172.21.15.29:0/460693982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e800a81b0 0x7f1e800a8590 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0097a0 tx=0x7f1e7c02b8b0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 -- 172.21.15.29:0/460693982 shutdown_connections 2023-08-22T23:44:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 --2- 172.21.15.29:0/460693982 >> v2:172.21.15.52:3300/0 conn(0x7f1e8013eba0 0x7f1e80144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 --2- 172.21.15.29:0/460693982 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e800a81b0 0x7f1e800a8590 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 -- 172.21.15.29:0/460693982 >> 172.21.15.29:0/460693982 conn(0x7f1e80065fb0 msgr2=0x7f1e800b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 -- 172.21.15.29:0/460693982 shutdown_connections 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.442+0000 7f1e873e8640 1 -- 172.21.15.29:0/460693982 wait complete. 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 10 monclient: build_initial_monmap 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 10 monclient: monmap: 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 1 librados: starting msgr at 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 1 librados: starting objecter 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 1 Processor -- start 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 1 -- start start 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 1 librados: setting wanted keys 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 1 librados: calling monclient init 2023-08-22T23:44:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.443+0000 7f1e873e8640 10 monclient: init 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e800a81b0 0x7f1e8013d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1e800a81b0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e8013e760 0x7f1e800aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1e8013e760 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e80142f70 0x7f1e80143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): picked mon.noname-b con 0x7f1e80142f70 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8013dd30 con 0x7f1e800a81b0 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e80143b10 con 0x7f1e80142f70 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e80144370 con 0x7f1e8013e760 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e873e8640 10 monclient(hunting): authenticate will time out at 2851.694837s 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e84983640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e8013e760 0x7f1e800aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.444+0000 7f1e85184640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e800a81b0 0x7f1e8013d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e8013e760 0x7f1e800aa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34336/0 (socket says 172.21.15.29:34336) 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 1 -- 172.21.15.29:0/2461810006 learned_addr learned my addr 172.21.15.29:0/2461810006 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): get_auth_request con 0x7f1e8013e760 auth_method 0 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85985640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e80142f70 0x7f1e80143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): get_auth_request con 0x7f1e800a81b0 auth_method 0 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85985640 10 monclient(hunting): get_auth_request con 0x7f1e80142f70 auth_method 0 2023-08-22T23:44:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85985640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85985640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85985640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e84983640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.445+0000 7f1e85184640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.446+0000 7f1e85985640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.446+0000 7f1e85985640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.446+0000 7f1e85985640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 10 monclient(hunting): handle_auth_done global_id 16709 payload 824 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 1 -- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e80142f70 msgr2=0x7f1e80143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e80142f70 0x7f1e80143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 1 -- 172.21.15.29:0/2461810006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e800a81b0 msgr2=0x7f1e8013d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e800a81b0 0x7f1e8013d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 10 monclient: _finish_hunting 0 2023-08-22T23:44:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 1 monclient: found mon.noname-c 2023-08-22T23:44:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.448+0000 7f1e84983640 1 -- 172.21.15.29:0/2461810006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c009400 con 0x7f1e8013e760 2023-08-22T23:44:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.450+0000 7f1e84983640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e8013e760 0x7f1e800aa6a0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000c460 tx=0x7f1e7000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.450+0000 7f1e75ffb640 10 client.?.objecter ms_handle_connect 0x7f1e8013e760 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.450+0000 7f1e75ffb640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70005ce0 con 0x7f1e8013e760 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: dump: 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.463 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: _finish_auth 0 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: _check_auth_tickets 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70005e40 con 0x7f1e8013e760 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70007290 con 0x7f1e8013e760 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e75ffb640 10 monclient: dump: 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 5 monclient: authenticate success, global_id 16709 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 10 monclient: _renew_subs 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e777fe640 4 set_mon_vals no callback set 2023-08-22T23:44:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e80144650 con 0x7f1e8013e760 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 10 client.16709.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 10 monclient: _renew_subs 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e800aabe0 con 0x7f1e8013e760 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.454+0000 7f1e873e8640 1 librados: init done 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.455+0000 7f1e873e8640 10 client.16709.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.455+0000 7f1e873e8640 15 client.16709.objecter _session_command_op_assign -1 1 2023-08-22T23:44:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.455+0000 7f1e873e8640 20 client.16709.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.455+0000 7f1e873e8640 10 client.16709.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.455+0000 7f1e873e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.455+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e800b0bc0 con 0x7f1e8013e760 2023-08-22T23:44:47.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.474+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1e700073f0 con 0x7f1e8013e760 2023-08-22T23:44:47.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.474+0000 7f1e75ffb640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1e6406ba60 0x7f1e6406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.475+0000 7f1e85184640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1e6406ba60 0x7f1e6406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.475+0000 7f1e85184640 10 monclient: get_auth_request con 0x7f1e6406ba60 auth_method 0 2023-08-22T23:44:47.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.475+0000 7f1e85184640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1e6406ba60 0x7f1e6406df20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c036000 tx=0x7f1e7c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.475+0000 7f1e75ffb640 10 client.16709.objecter ms_handle_connect 0x7f1e6406ba60 2023-08-22T23:44:47.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.476+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f1e7005b070 con 0x7f1e8013e760 2023-08-22T23:44:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.476+0000 7f1e75ffb640 10 client.16709.objecter ms_dispatch 0x7f1e80000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.476+0000 7f1e75ffb640 3 client.16709.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.476+0000 7f1e75ffb640 3 client.16709.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 10 client.16709.objecter checking command 1 2023-08-22T23:44:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f1e640731d0 0x7f1e64075670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 20 client.16709.objecter _get_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 15 client.16709.objecter _session_command_op_remove -1 1 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 20 client.16709.objecter _get_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 20 client.16709.objecter get_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 15 client.16709.objecter _session_command_op_assign 4 1 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 4 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 10 client.16709.objecter _send_command 1 2023-08-22T23:44:47.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 --> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e64075d40 con 0x7f1e640731d0 2023-08-22T23:44:47.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 20 client.16709.objecter dump_active .. 0 homeless 2023-08-22T23:44:47.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e85985640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f1e640731d0 0x7f1e64075670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e700536a0 con 0x7f1e8013e760 2023-08-22T23:44:47.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.477+0000 7f1e75ffb640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:47.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.478+0000 7f1e85985640 10 monclient: get_auth_request con 0x7f1e640731d0 auth_method 0 2023-08-22T23:44:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.481+0000 7f1e75ffb640 10 client.16709.objecter ms_handle_connect 0x7f1e640731d0 2023-08-22T23:44:47.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.481+0000 7f1e85985640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f1e640731d0 0x7f1e64075670 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.506+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== osd.4 v2:172.21.15.52:6802/2157117708 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1e64075d40 con 0x7f1e640731d0 2023-08-22T23:44:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.506+0000 7f1e75ffb640 10 client.16709.objecter ms_dispatch 0x7f1e80000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.506+0000 7f1e75ffb640 10 client.16709.objecter _finish_command 1 = system:0 2023-08-22T23:44:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.506+0000 7f1e75ffb640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 4 2023-08-22T23:44:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.506+0000 7f1e75ffb640 15 client.16709.objecter _session_command_op_remove 4 1 2023-08-22T23:44:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 10 client.16709.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 15 client.16709.objecter _session_command_op_assign -1 2 2023-08-22T23:44:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 20 client.16709.objecter _get_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 20 client.16709.objecter _get_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 15 client.16709.objecter _session_command_op_remove -1 2 2023-08-22T23:44:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 20 client.16709.objecter get_session s=0x7f1e64072b00 osd=4 3 2023-08-22T23:44:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 15 client.16709.objecter _session_command_op_assign 4 2 2023-08-22T23:44:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 4 2023-08-22T23:44:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 10 client.16709.objecter _send_command 2 2023-08-22T23:44:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.519+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 --> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1e800a9df0 con 0x7f1e640731d0 2023-08-22T23:44:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e75ffb640 1 -- 172.21.15.29:0/2461810006 <== osd.4 v2:172.21.15.52:6802/2157117708 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1e7800f040 con 0x7f1e640731d0 2023-08-22T23:44:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e75ffb640 10 client.16709.objecter ms_dispatch 0x7f1e80000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e75ffb640 10 client.16709.objecter _finish_command 2 = system:0 2023-08-22T23:44:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e75ffb640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 4 2023-08-22T23:44:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e75ffb640 15 client.16709.objecter _session_command_op_remove 4 2 2023-08-22T23:44:47.537 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:47.537 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:47.538 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 10 librados: watch_flush enter 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 10 librados: watch_flush exit 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 10 client.16709.objecter close_session for osd.4 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f1e640731d0 msgr2=0x7f1e64075670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6802/2157117708,v1:172.21.15.52:6803/2157117708] conn(0x7f1e640731d0 0x7f1e64075670 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 20 client.16709.objecter put_session s=0x7f1e64072b00 osd=4 1 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 20 client.16709.objecter shutdown clearing up homeless session... 2023-08-22T23:44:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 10 client.16709.objecter successfully canceled tick 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1e6406ba60 msgr2=0x7f1e6406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f1e6406ba60 0x7f1e6406df20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c036000 tx=0x7f1e7c03c040 comp rx=0 tx=0).stop 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 10 monclient: shutdown 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e8013e760 msgr2=0x7f1e800aa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 --2- 172.21.15.29:0/2461810006 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f1e8013e760 0x7f1e800aa6a0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000c460 tx=0x7f1e7000c920 comp rx=0 tx=0).stop 2023-08-22T23:44:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e85985640 1 -- 172.21.15.29:0/2461810006 reap_dead start 2023-08-22T23:44:47.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 shutdown_connections 2023-08-22T23:44:47.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 >> 172.21.15.29:0/2461810006 conn(0x7f1e80065fb0 msgr2=0x7f1e800aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:47.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 shutdown_connections 2023-08-22T23:44:47.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.535+0000 7f1e873e8640 1 -- 172.21.15.29:0/2461810006 wait complete. 2023-08-22T23:44:47.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.536+0000 7f1e873e8640 1 librados: shutdown 2023-08-22T23:44:47.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-08-22T23:44:47.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.653+0000 7fdde66cc640 10 monclient: get_monmap_and_config 2023-08-22T23:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.653+0000 7fdde66cc640 10 monclient: build_initial_monmap 2023-08-22T23:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.653+0000 7fdde66cc640 10 monclient: monmap: 2023-08-22T23:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.655 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:47.654945+0000 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:47.654945+0000 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:47.657 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 1 Processor -- start 2023-08-22T23:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 1 -- start start 2023-08-22T23:44:47.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 10 monclient: init 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 0x7fdde00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 10 monclient(hunting): picked mon.noname-a con 0x7fdde00adee0 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.654+0000 7fdde66cc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 0x7fdde013c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): picked mon.noname-b con 0x7fdde00ae2c0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fdde013c870 0x7fdde013f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): picked mon.noname-c con 0x7fdde013c870 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde00a7f90 con 0x7fdde00adee0 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde013eed0 con 0x7fdde013c870 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde0142430 con 0x7fdde00ae2c0 2023-08-22T23:44:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde66cc640 10 monclient(hunting): authenticate will time out at 2851.905505s 2023-08-22T23:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 0x7fdde00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddf7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 0x7fdde013c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdde4c69640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fdde013c870 0x7fdde013f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 0x7fdde00710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52712/0 (socket says 172.21.15.29:52712) 2023-08-22T23:44:47.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 1 -- 172.21.15.29:0/1062211852 learned_addr learned my addr 172.21.15.29:0/1062211852 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:47.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddf7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 0x7fdde013c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34352/0 (socket says 172.21.15.29:34352) 2023-08-22T23:44:47.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 10 monclient(hunting): get_auth_request con 0x7fdde00adee0 auth_method 0 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddf7fe640 10 monclient(hunting): get_auth_request con 0x7fdde00ae2c0 auth_method 0 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddf7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.655+0000 7fdddf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): get_auth_request con 0x7fdde013c870 auth_method 0 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.656+0000 7fdde4c69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_done global_id 16712 payload 274 2023-08-22T23:44:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1062211852 >> v2:172.21.15.52:3300/0 conn(0x7fdde013c870 msgr2=0x7fdde013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 1 --2- 172.21.15.29:0/1062211852 >> v2:172.21.15.52:3300/0 conn(0x7fdde013c870 0x7fdde013f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1062211852 >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 msgr2=0x7fdde00710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 1 --2- 172.21.15.29:0/1062211852 >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 0x7fdde00710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 1 monclient: found mon.noname-b 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1062211852 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde01425a0 con 0x7fdde00ae2c0 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.658+0000 7fdddffff640 1 --2- 172.21.15.29:0/1062211852 >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 0x7fdde00710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.660+0000 7fdddf7fe640 1 --2- 172.21.15.29:0/1062211852 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 0x7fdde013c330 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fdde006bad0 tx=0x7fddd002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f90a2d3db21b9f83 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 1 -- 172.21.15.29:0/1062211852 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0030710 con 0x7fdde00ae2c0 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: dump: 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.672 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.673 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: _finish_auth 0 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: _check_auth_tickets 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 1 -- 172.21.15.29:0/1062211852 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd002e050 con 0x7fdde00ae2c0 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 1 -- 172.21.15.29:0/1062211852 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0030a90 con 0x7fdde00ae2c0 2023-08-22T23:44:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdde66cc640 5 monclient: authenticate success, global_id 16712 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdde66cc640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdde66cc640 4 set_mon_vals no callback set 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.663+0000 7fdddeffd640 10 monclient: dump: 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 10 monclient: shutdown 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 -- 172.21.15.29:0/1062211852 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 msgr2=0x7fdde013c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1062211852 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 0x7fdde013c330 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fdde006bad0 tx=0x7fddd002b710 comp rx=0 tx=0).stop 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 -- 172.21.15.29:0/1062211852 shutdown_connections 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1062211852 >> v2:172.21.15.52:3300/0 conn(0x7fdde013c870 0x7fdde013f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1062211852 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde00ae2c0 0x7fdde013c330 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1062211852 >> v2:172.21.15.29:3300/0 conn(0x7fdde00adee0 0x7fdde00710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 -- 172.21.15.29:0/1062211852 >> 172.21.15.29:0/1062211852 conn(0x7fdde00afad0 msgr2=0x7fdde00b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 -- 172.21.15.29:0/1062211852 shutdown_connections 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.664+0000 7fdde66cc640 1 -- 172.21.15.29:0/1062211852 wait complete. 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 10 monclient: build_initial_monmap 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 10 monclient: monmap: 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 1 librados: starting msgr at 2023-08-22T23:44:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 1 librados: starting objecter 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 1 Processor -- start 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 1 -- start start 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 1 librados: setting wanted keys 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 1 librados: calling monclient init 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.665+0000 7fdde66cc640 10 monclient: init 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdde00adee0 0x7fdde0141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): picked mon.noname-b con 0x7fdde00adee0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 0x7fdde01d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): picked mon.noname-c con 0x7fdde013c870 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdde01d76f0 0x7fdde0137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): picked mon.noname-a con 0x7fdde01d76f0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00abdf0 con 0x7fdde01d76f0 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01d7b70 con 0x7fdde00adee0 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01d7d90 con 0x7fdde013c870 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.666+0000 7fdde66cc640 10 monclient(hunting): authenticate will time out at 2851.917030s 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 0x7fdde01d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdde01d76f0 0x7fdde0137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 0x7fdde01d71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34364/0 (socket says 172.21.15.29:34364) 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1867011658 learned_addr learned my addr 172.21.15.29:0/1867011658 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): get_auth_request con 0x7fdde013c870 auth_method 0 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): get_auth_request con 0x7fdde01d76f0 auth_method 0 2023-08-22T23:44:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddffff640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdde00adee0 0x7fdde0141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddffff640 10 monclient(hunting): get_auth_request con 0x7fdde00adee0 auth_method 0 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.667+0000 7fdde4c69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.668+0000 7fdddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.668+0000 7fdddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.668+0000 7fdddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 10 monclient(hunting): handle_auth_done global_id 16715 payload 824 2023-08-22T23:44:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdde00adee0 msgr2=0x7fdde0141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdde00adee0 0x7fdde0141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdde01d76f0 msgr2=0x7fdde0137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdde01d76f0 0x7fdde0137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 1 monclient: found mon.noname-c 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.670+0000 7fdddf7fe640 1 -- 172.21.15.29:0/1867011658 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd40094d0 con 0x7fdde013c870 2023-08-22T23:44:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.673+0000 7fdddf7fe640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 0x7fdde01d71b0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009860 tx=0x7fddd00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.673+0000 7fdddcff9640 10 client.?.objecter ms_handle_connect 0x7fdde013c870 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.673+0000 7fdddcff9640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd003c6e0 con 0x7fdde013c870 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: dump: 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.690 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: _finish_auth 0 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: _check_auth_tickets 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd003c840 con 0x7fdde013c870 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 5 monclient: authenticate success, global_id 16715 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 10 monclient: _renew_subs 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd0009400 con 0x7fdde013c870 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd003cbf0 con 0x7fdde013c870 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdddcff9640 10 monclient: dump: 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fddde7fc640 4 set_mon_vals no callback set 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 10 client.16715.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 10 monclient: _renew_subs 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde01379f0 con 0x7fdde013c870 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.676+0000 7fdde66cc640 1 librados: init done 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.677+0000 7fdde66cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:47.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.677+0000 7fdde66cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.677+0000 7fdde66cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.677+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde01d7fb0 con 0x7fdde013c870 2023-08-22T23:44:47.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.947+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fddd0039070 con 0x7fdde013c870 2023-08-22T23:44:47.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.948+0000 7fdddcff9640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fddcc06ba30 0x7fddcc06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:47.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.948+0000 7fdddffff640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fddcc06ba30 0x7fddcc06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:47.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.948+0000 7fdddffff640 10 monclient: get_auth_request con 0x7fddcc06ba30 auth_method 0 2023-08-22T23:44:47.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.949+0000 7fdddffff640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fddcc06ba30 0x7fddcc06def0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006ca0 tx=0x7fddd4009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:47.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.949+0000 7fdddcff9640 10 client.16715.objecter ms_handle_connect 0x7fddcc06ba30 2023-08-22T23:44:47.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fddd007d220 con 0x7fdde013c870 2023-08-22T23:44:47.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 10 client.16715.objecter ms_dispatch 0x7fdde0000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 3 client.16715.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 3 client.16715.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 20 client.16715.objecter dump_active .. 0 homeless 2023-08-22T23:44:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fddd0075850 con 0x7fdde013c870 2023-08-22T23:44:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:47.960+0000 7fdddcff9640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:48.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.143+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdde0137c10 con 0x7fddcc06ba30 2023-08-22T23:44:48.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.144+0000 7fdddcff9640 1 -- 172.21.15.29:0/1867011658 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdde0137c10 con 0x7fddcc06ba30 2023-08-22T23:44:48.145 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 4 to deep-scrub 2023-08-22T23:44:48.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 10 librados: watch_flush enter 2023-08-22T23:44:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 10 librados: watch_flush exit 2023-08-22T23:44:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 20 client.16715.objecter shutdown clearing up homeless session... 2023-08-22T23:44:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 10 client.16715.objecter successfully canceled tick 2023-08-22T23:44:48.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fddcc06ba30 msgr2=0x7fddcc06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fddcc06ba30 0x7fddcc06def0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006ca0 tx=0x7fddd4009240 comp rx=0 tx=0).stop 2023-08-22T23:44:48.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 10 monclient: shutdown 2023-08-22T23:44:48.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 msgr2=0x7fdde01d71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 0x7fdde01d71b0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009860 tx=0x7fddd00046c0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 shutdown_connections 2023-08-22T23:44:48.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fddcc06ba30 0x7fddcc06def0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdde01d76f0 0x7fdde0137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fdde013c870 0x7fdde01d71b0 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 --2- 172.21.15.29:0/1867011658 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdde00adee0 0x7fdde0141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.145+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 >> 172.21.15.29:0/1867011658 conn(0x7fdde00afad0 msgr2=0x7fdde0070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:48.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.146+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 shutdown_connections 2023-08-22T23:44:48.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.146+0000 7fdde66cc640 1 -- 172.21.15.29:0/1867011658 wait complete. 2023-08-22T23:44:48.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.146+0000 7fdde66cc640 1 librados: shutdown 2023-08-22T23:44:48.157 INFO:tasks.ceph:Scrubbing osd.5 2023-08-22T23:44:48.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.270+0000 7f53e55a4640 10 monclient: get_monmap_and_config 2023-08-22T23:44:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.270+0000 7f53e55a4640 10 monclient: build_initial_monmap 2023-08-22T23:44:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.270+0000 7f53e55a4640 10 monclient: monmap: 2023-08-22T23:44:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:48.271382+0000 2023-08-22T23:44:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:48.271382+0000 2023-08-22T23:44:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:48.273 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.270+0000 7f53e55a4640 1 Processor -- start 2023-08-22T23:44:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.270+0000 7f53e55a4640 1 -- start start 2023-08-22T23:44:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.270+0000 7f53e55a4640 10 monclient: init 2023-08-22T23:44:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-08-22T23:44:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53e00a81b0 0x7f53e00a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): picked mon.noname-a con 0x7f53e00a81b0 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f53e013e760 0x7f53e0144720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): picked mon.noname-c con 0x7f53e013e760 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53e0144c60 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e00b07e0 con 0x7f53e0144c60 2023-08-22T23:44:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e00a7d70 con 0x7f53e00a81b0 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e00a98a0 con 0x7f53e013e760 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53e55a4640 10 monclient(hunting): authenticate will time out at 2852.521869s 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.271+0000 7f53deffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53e00a81b0 0x7f53e00a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53df7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f53e0144c60 0x7f53e0141190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48142/0 (socket says 172.21.15.29:48142) 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53df7fe640 1 -- 172.21.15.29:0/2037454001 learned_addr learned my addr 172.21.15.29:0/2037454001 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53de7fc640 1 --2- 172.21.15.29:0/2037454001 >> v2:172.21.15.52:3300/0 conn(0x7f53e013e760 0x7f53e0144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53deffd640 10 monclient(hunting): get_auth_request con 0x7f53e00a81b0 auth_method 0 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53deffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53deffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53de7fc640 10 monclient(hunting): get_auth_request con 0x7f53e013e760 auth_method 0 2023-08-22T23:44:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53de7fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.272+0000 7f53de7fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.273+0000 7f53de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.273+0000 7f53de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.273+0000 7f53de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.273+0000 7f53deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.273+0000 7f53deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.273+0000 7f53deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 10 monclient(hunting): handle_auth_done global_id 19624 payload 274 2023-08-22T23:44:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 1 -- 172.21.15.29:0/2037454001 >> v2:172.21.15.29:3300/0 conn(0x7f53e00a81b0 msgr2=0x7f53e00a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 1 --2- 172.21.15.29:0/2037454001 >> v2:172.21.15.29:3300/0 conn(0x7f53e00a81b0 0x7f53e00a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 1 -- 172.21.15.29:0/2037454001 >> v1:172.21.15.29:6789/0 conn(0x7f53e0144c60 legacy=0x7f53e0141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 10 monclient: _finish_hunting 0 2023-08-22T23:44:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 1 monclient: found mon.noname-c 2023-08-22T23:44:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.274+0000 7f53de7fc640 1 -- 172.21.15.29:0/2037454001 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00a8ad0 con 0x7f53e013e760 2023-08-22T23:44:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.276+0000 7f53de7fc640 1 --2- 172.21.15.29:0/2037454001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e013e760 0x7f53e0144720 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 crypto rx=0x7f53e00a7600 tx=0x7f53c802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb7430da849ed35 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 1 -- 172.21.15.29:0/2037454001 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8031830 con 0x7f53e013e760 2023-08-22T23:44:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: dump: 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.287 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: _finish_auth 0 2023-08-22T23:44:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: _check_auth_tickets 2023-08-22T23:44:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 1 -- 172.21.15.29:0/2037454001 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c8031990 con 0x7f53e013e760 2023-08-22T23:44:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:48.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 1 -- 172.21.15.29:0/2037454001 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8031d20 con 0x7f53e013e760 2023-08-22T23:44:48.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53e55a4640 5 monclient: authenticate success, global_id 19624 2023-08-22T23:44:48.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53e55a4640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:48.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53e55a4640 4 set_mon_vals no callback set 2023-08-22T23:44:48.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:48.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.280+0000 7f53ddffb640 10 monclient: dump: 2023-08-22T23:44:48.291 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.291 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.291 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.291 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.292 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.292 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.292 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.293 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.293 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.293 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 10 monclient: shutdown 2023-08-22T23:44:48.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 -- 172.21.15.29:0/2037454001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e013e760 msgr2=0x7f53e0144720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 --2- 172.21.15.29:0/2037454001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e013e760 0x7f53e0144720 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 crypto rx=0x7f53e00a7600 tx=0x7f53c802b8d0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 -- 172.21.15.29:0/2037454001 shutdown_connections 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 --2- 172.21.15.29:0/2037454001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e013e760 0x7f53e0144720 unknown :-1 s=CLOSED pgs=4632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 --2- 172.21.15.29:0/2037454001 >> v2:172.21.15.29:3300/0 conn(0x7f53e00a81b0 0x7f53e00a8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 -- 172.21.15.29:0/2037454001 >> 172.21.15.29:0/2037454001 conn(0x7f53e0065fb0 msgr2=0x7f53e00b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 -- 172.21.15.29:0/2037454001 shutdown_connections 2023-08-22T23:44:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.281+0000 7f53e55a4640 1 -- 172.21.15.29:0/2037454001 wait complete. 2023-08-22T23:44:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 10 monclient: build_initial_monmap 2023-08-22T23:44:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 10 monclient: monmap: 2023-08-22T23:44:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:48.297 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 1 librados: starting msgr at 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 1 librados: starting objecter 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 1 Processor -- start 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 1 -- start start 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 1 librados: setting wanted keys 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 1 librados: calling monclient init 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.282+0000 7f53e55a4640 10 monclient: init 2023-08-22T23:44:48.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:44:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53e00a81b0 0x7f53e00acd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): picked mon.noname-a con 0x7f53e00a81b0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f53e013e760 0x7f53e0139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): picked mon.noname-c con 0x7f53e013e760 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e0144c60 0x7f53e013a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): picked mon.noname-b con 0x7f53e0144c60 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e013f880 con 0x7f53e00a81b0 2023-08-22T23:44:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0141100 con 0x7f53e0144c60 2023-08-22T23:44:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e013a570 con 0x7f53e013e760 2023-08-22T23:44:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:48.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53e55a4640 10 monclient(hunting): authenticate will time out at 2852.533718s 2023-08-22T23:44:48.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53deffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53e00a81b0 0x7f53e00acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.283+0000 7f53de7fc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f53e013e760 0x7f53e0139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53e00a81b0 0x7f53e00acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52748/0 (socket says 172.21.15.29:52748) 2023-08-22T23:44:48.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 1 -- 172.21.15.29:0/1227299811 learned_addr learned my addr 172.21.15.29:0/1227299811 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:48.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): get_auth_request con 0x7f53e00a81b0 auth_method 0 2023-08-22T23:44:48.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e0144c60 0x7f53e013a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): get_auth_request con 0x7f53e013e760 auth_method 0 2023-08-22T23:44:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): get_auth_request con 0x7f53e0144c60 auth_method 0 2023-08-22T23:44:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.284+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 10 monclient(hunting): handle_auth_done global_id 16718 payload 824 2023-08-22T23:44:48.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 1 -- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e0144c60 msgr2=0x7f53e013a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53e0144c60 0x7f53e013a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 1 -- 172.21.15.29:0/1227299811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53e00a81b0 msgr2=0x7f53e00acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53e00a81b0 0x7f53e00acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 10 monclient: _finish_hunting 0 2023-08-22T23:44:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 1 monclient: found mon.noname-c 2023-08-22T23:44:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:44:48.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53de7fc640 1 -- 172.21.15.29:0/1227299811 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c8009400 con 0x7f53e013e760 2023-08-22T23:44:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.287+0000 7f53deffd640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53e00a81b0 0x7f53e00acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:44:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.289+0000 7f53de7fc640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f53e013e760 0x7f53e0139af0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f53c8033000 tx=0x7f53c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.289+0000 7f53d77fe640 10 client.?.objecter ms_handle_connect 0x7f53e013e760 2023-08-22T23:44:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.289+0000 7f53d77fe640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:48.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c803a6f0 con 0x7f53e013e760 2023-08-22T23:44:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: dump: 2023-08-22T23:44:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.313 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.314 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: _finish_auth 0 2023-08-22T23:44:48.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: _check_auth_tickets 2023-08-22T23:44:48.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:48.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c803a870 con 0x7f53e013e760 2023-08-22T23:44:48.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 5 monclient: authenticate success, global_id 16718 2023-08-22T23:44:48.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 10 monclient: _renew_subs 2023-08-22T23:44:48.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:48.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e00ad240 con 0x7f53e013e760 2023-08-22T23:44:48.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:48.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c803ac20 con 0x7f53e013e760 2023-08-22T23:44:48.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:48.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53d77fe640 10 monclient: dump: 2023-08-22T23:44:48.318 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.318 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.318 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.318 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.320 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53dd7fa640 4 set_mon_vals no callback set 2023-08-22T23:44:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 10 client.16718.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 10 monclient: _renew_subs 2023-08-22T23:44:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e013f450 con 0x7f53e013e760 2023-08-22T23:44:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.293+0000 7f53e55a4640 1 librados: init done 2023-08-22T23:44:48.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.294+0000 7f53e55a4640 10 client.16718.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:48.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.294+0000 7f53e55a4640 15 client.16718.objecter _session_command_op_assign -1 1 2023-08-22T23:44:48.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.294+0000 7f53e55a4640 20 client.16718.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:48.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.294+0000 7f53e55a4640 10 client.16718.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:48.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.294+0000 7f53e55a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:44:48.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.294+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53e013f700 con 0x7f53e013e760 2023-08-22T23:44:48.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.313+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f53c804c3b0 con 0x7f53e013e760 2023-08-22T23:44:48.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.314+0000 7f53d77fe640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f53c006ba30 0x7f53c006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.314+0000 7f53deffd640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f53c006ba30 0x7f53c006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.314+0000 7f53deffd640 10 monclient: get_auth_request con 0x7f53c006ba30 auth_method 0 2023-08-22T23:44:48.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.315+0000 7f53d77fe640 10 client.16718.objecter ms_handle_connect 0x7f53c006ba30 2023-08-22T23:44:48.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.315+0000 7f53deffd640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f53c006ba30 0x7f53c006def0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f53d80112c0 tx=0x7f53d8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f53c807dce0 con 0x7f53e013e760 2023-08-22T23:44:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 10 client.16718.objecter ms_dispatch 0x7f53e0000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 3 client.16718.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:48.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 3 client.16718.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 10 client.16718.objecter checking command 1 2023-08-22T23:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f53c0073230 0x7f53c00756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 20 client.16718.objecter _get_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 15 client.16718.objecter _session_command_op_remove -1 1 2023-08-22T23:44:48.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 20 client.16718.objecter _get_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 20 client.16718.objecter get_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 15 client.16718.objecter _session_command_op_assign 5 1 2023-08-22T23:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 4 2023-08-22T23:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 10 client.16718.objecter _send_command 1 2023-08-22T23:44:48.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 --> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53c0075da0 con 0x7f53c0073230 2023-08-22T23:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.316+0000 7f53d77fe640 20 client.16718.objecter dump_active .. 0 homeless 2023-08-22T23:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.317+0000 7f53df7fe640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f53c0073230 0x7f53c00756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.317+0000 7f53df7fe640 10 monclient: get_auth_request con 0x7f53c0073230 auth_method 0 2023-08-22T23:44:48.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.317+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53c8076310 con 0x7f53e013e760 2023-08-22T23:44:48.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.317+0000 7f53d77fe640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:48.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.320+0000 7f53d77fe640 10 client.16718.objecter ms_handle_connect 0x7f53c0073230 2023-08-22T23:44:48.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.320+0000 7f53df7fe640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f53c0073230 0x7f53c00756d0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.345+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== osd.5 v2:172.21.15.52:6826/4164832567 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f53c0075da0 con 0x7f53c0073230 2023-08-22T23:44:48.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.345+0000 7f53d77fe640 10 client.16718.objecter ms_dispatch 0x7f53e0000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:48.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.345+0000 7f53d77fe640 10 client.16718.objecter _finish_command 1 = system:0 2023-08-22T23:44:48.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.345+0000 7f53d77fe640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 4 2023-08-22T23:44:48.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.345+0000 7f53d77fe640 15 client.16718.objecter _session_command_op_remove 5 1 2023-08-22T23:44:48.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 10 client.16718.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:48.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 15 client.16718.objecter _session_command_op_assign -1 2 2023-08-22T23:44:48.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 20 client.16718.objecter _get_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 20 client.16718.objecter _get_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 15 client.16718.objecter _session_command_op_remove -1 2 2023-08-22T23:44:48.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 20 client.16718.objecter get_session s=0x7f53c0072b60 osd=5 3 2023-08-22T23:44:48.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 15 client.16718.objecter _session_command_op_assign 5 2 2023-08-22T23:44:48.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 4 2023-08-22T23:44:48.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 10 client.16718.objecter _send_command 2 2023-08-22T23:44:48.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.358+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 --> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f53e00a9df0 con 0x7f53c0073230 2023-08-22T23:44:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.373+0000 7f53d77fe640 1 -- 172.21.15.29:0/1227299811 <== osd.5 v2:172.21.15.52:6826/4164832567 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f53d0001e60 con 0x7f53c0073230 2023-08-22T23:44:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.373+0000 7f53d77fe640 10 client.16718.objecter ms_dispatch 0x7f53e0000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.373+0000 7f53d77fe640 10 client.16718.objecter _finish_command 2 = system:0 2023-08-22T23:44:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.373+0000 7f53d77fe640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 4 2023-08-22T23:44:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.373+0000 7f53d77fe640 15 client.16718.objecter _session_command_op_remove 5 2 2023-08-22T23:44:48.375 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:48.376 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:48.376 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 10 librados: watch_flush enter 2023-08-22T23:44:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 10 librados: watch_flush exit 2023-08-22T23:44:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 10 client.16718.objecter close_session for osd.5 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f53c0073230 msgr2=0x7f53c00756d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6826/4164832567,v1:172.21.15.52:6827/4164832567] conn(0x7f53c0073230 0x7f53c00756d0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 20 client.16718.objecter put_session s=0x7f53c0072b60 osd=5 1 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 20 client.16718.objecter shutdown clearing up homeless session... 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 10 client.16718.objecter successfully canceled tick 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f53c006ba30 msgr2=0x7f53c006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f53c006ba30 0x7f53c006def0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f53d80112c0 tx=0x7f53d8009190 comp rx=0 tx=0).stop 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 10 monclient: shutdown 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f53e013e760 msgr2=0x7f53e0139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 --2- 172.21.15.29:0/1227299811 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f53e013e760 0x7f53e0139af0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f53c8033000 tx=0x7f53c80046c0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53df7fe640 1 -- 172.21.15.29:0/1227299811 reap_dead start 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 shutdown_connections 2023-08-22T23:44:48.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 >> 172.21.15.29:0/1227299811 conn(0x7f53e0065fb0 msgr2=0x7f53e013df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:48.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 shutdown_connections 2023-08-22T23:44:48.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 -- 172.21.15.29:0/1227299811 wait complete. 2023-08-22T23:44:48.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.374+0000 7f53e55a4640 1 librados: shutdown 2023-08-22T23:44:48.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-08-22T23:44:48.452 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:33.736 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.499 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:33.797 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.501 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:33.799 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.502 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:33.786 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.508+0000 7fe84af82640 10 monclient: get_monmap_and_config 2023-08-22T23:44:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.509+0000 7fe84af82640 10 monclient: build_initial_monmap 2023-08-22T23:44:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.509+0000 7fe84af82640 10 monclient: monmap: 2023-08-22T23:44:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:48.510327+0000 2023-08-22T23:44:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:48.510327+0000 2023-08-22T23:44:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:48.513 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.509+0000 7fe84af82640 1 Processor -- start 2023-08-22T23:44:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.509+0000 7fe84af82640 1 -- start start 2023-08-22T23:44:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.509+0000 7fe84af82640 10 monclient: init 2023-08-22T23:44:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8441448e0 0x7fe844144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): picked mon.noname-a con 0x7fe8441448e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fe8440add90 0x7fe844145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): picked mon.noname-c con 0x7fe8440add90 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe84413e190 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8440b12d0 con 0x7fe84413e190 2023-08-22T23:44:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440afae0 con 0x7fe8441448e0 2023-08-22T23:44:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440afdc0 con 0x7fe8440add90 2023-08-22T23:44:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.510+0000 7fe84af82640 10 monclient(hunting): authenticate will time out at 2852.761134s 2023-08-22T23:44:48.517 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:33.812 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.515+0000 7fe84951f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe84413e190 0x7fe8441410f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48144/0 (socket says 172.21.15.29:48144) 2023-08-22T23:44:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.515+0000 7fe84951f640 1 -- 172.21.15.29:0/2259255750 learned_addr learned my addr 172.21.15.29:0/2259255750 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.516+0000 7fe848d1e640 1 --2- 172.21.15.29:0/2259255750 >> v2:172.21.15.29:3300/0 conn(0x7fe8441448e0 0x7fe844144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.517+0000 7fe848d1e640 10 monclient(hunting): get_auth_request con 0x7fe8441448e0 auth_method 0 2023-08-22T23:44:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.517+0000 7fe848d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.517+0000 7fe848d1e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.517+0000 7fe848d1e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.519+0000 7fe83bfff640 1 --2- 172.21.15.29:0/2259255750 >> v2:172.21.15.52:3300/0 conn(0x7fe8440add90 0x7fe844145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe848d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe848d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe848d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe83b7fe640 1 -- 172.21.15.29:0/2259255750 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869566552 0 0) 0x7fe8440b12d0 con 0x7fe84413e190 2023-08-22T23:44:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe83b7fe640 10 monclient(hunting): _negotiate 2023-08-22T23:44:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe83b7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe83b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe83b7fe640 10 monclient(hunting): my global_id is 14811 2023-08-22T23:44:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.520+0000 7fe83b7fe640 1 -- 172.21.15.29:0/2259255750 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe82c003520 con 0x7fe84413e190 2023-08-22T23:44:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.522+0000 7fe83bfff640 10 monclient(hunting): get_auth_request con 0x7fe8440add90 auth_method 0 2023-08-22T23:44:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.522+0000 7fe83bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.522+0000 7fe83bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.522+0000 7fe83bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.524+0000 7fe83bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.524+0000 7fe83bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.524+0000 7fe83bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 10 monclient(hunting): handle_auth_done global_id 14814 payload 274 2023-08-22T23:44:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 1 -- 172.21.15.29:0/2259255750 >> v2:172.21.15.52:3300/0 conn(0x7fe8440add90 msgr2=0x7fe844145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 1 --2- 172.21.15.29:0/2259255750 >> v2:172.21.15.52:3300/0 conn(0x7fe8440add90 0x7fe844145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 1 -- 172.21.15.29:0/2259255750 >> v1:172.21.15.29:6789/0 conn(0x7fe84413e190 legacy=0x7fe8441410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 10 monclient: _finish_hunting 0 2023-08-22T23:44:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 1 monclient: found mon.noname-a 2023-08-22T23:44:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.525+0000 7fe848d1e640 1 -- 172.21.15.29:0/2259255750 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440b00a0 con 0x7fe8441448e0 2023-08-22T23:44:48.531 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:33.815 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.534+0000 7fe848d1e640 1 --2- 172.21.15.29:0/2259255750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8441448e0 0x7fe844144cc0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fe84413eec0 tx=0x7fe84002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d26114047d6123b server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 1 -- 172.21.15.29:0/2259255750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84002e070 con 0x7fe8441448e0 2023-08-22T23:44:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: dump: 2023-08-22T23:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: _finish_auth 0 2023-08-22T23:44:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: _check_auth_tickets 2023-08-22T23:44:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 1 -- 172.21.15.29:0/2259255750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8400315a0 con 0x7fe8441448e0 2023-08-22T23:44:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 1 -- 172.21.15.29:0/2259255750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840031930 con 0x7fe8441448e0 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe83b7fe640 10 monclient: dump: 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe84af82640 5 monclient: authenticate success, global_id 14814 2023-08-22T23:44:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe84af82640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.546+0000 7fe84af82640 4 set_mon_vals no callback set 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 10 monclient: shutdown 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 -- 172.21.15.29:0/2259255750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8441448e0 msgr2=0x7fe844144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 --2- 172.21.15.29:0/2259255750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8441448e0 0x7fe844144cc0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fe84413eec0 tx=0x7fe84002b840 comp rx=0 tx=0).stop 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 -- 172.21.15.29:0/2259255750 shutdown_connections 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 --2- 172.21.15.29:0/2259255750 >> v2:172.21.15.52:3300/0 conn(0x7fe8440add90 0x7fe844145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 --2- 172.21.15.29:0/2259255750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8441448e0 0x7fe844144cc0 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 -- 172.21.15.29:0/2259255750 >> 172.21.15.29:0/2259255750 conn(0x7fe844065ed0 msgr2=0x7fe8440b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 -- 172.21.15.29:0/2259255750 shutdown_connections 2023-08-22T23:44:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.547+0000 7fe84af82640 1 -- 172.21.15.29:0/2259255750 wait complete. 2023-08-22T23:44:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.548+0000 7fe84af82640 10 monclient: build_initial_monmap 2023-08-22T23:44:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.548+0000 7fe84af82640 10 monclient: monmap: 2023-08-22T23:44:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 1 librados: starting msgr at 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 1 librados: starting objecter 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 1 Processor -- start 2023-08-22T23:44:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 1 -- start start 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 1 librados: setting wanted keys 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 1 librados: calling monclient init 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.549+0000 7fe84af82640 10 monclient: init 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8440add90 0x7fe8441d5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): picked mon.noname-b con 0x7fe8440add90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 0x7fe8441d6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): picked mon.noname-a con 0x7fe84413e190 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe8441448e0 0x7fe8440a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): picked mon.noname-c con 0x7fe8441448e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8441d6250 con 0x7fe84413e190 2023-08-22T23:44:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe844141ac0 con 0x7fe8440add90 2023-08-22T23:44:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe844141c80 con 0x7fe8441448e0 2023-08-22T23:44:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.550+0000 7fe84af82640 10 monclient(hunting): authenticate will time out at 2852.801088s 2023-08-22T23:44:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.554+0000 7fe83bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 0x7fe8441d6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.556+0000 7fe83bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 0x7fe8441d6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48068/0 (socket says 172.21.15.29:48068) 2023-08-22T23:44:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.556+0000 7fe83bfff640 1 -- 172.21.15.29:0/3458509936 learned_addr learned my addr 172.21.15.29:0/3458509936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.556+0000 7fe83bfff640 10 monclient(hunting): get_auth_request con 0x7fe84413e190 auth_method 0 2023-08-22T23:44:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.556+0000 7fe83bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.556+0000 7fe83bfff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.556+0000 7fe83bfff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.564 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:33.842 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.557+0000 7fe84951f640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe8441448e0 0x7fe8440a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.558+0000 7fe848d1e640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8440add90 0x7fe8441d5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.558+0000 7fe83bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.558+0000 7fe83bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.558+0000 7fe83bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.559+0000 7fe84951f640 10 monclient(hunting): get_auth_request con 0x7fe8441448e0 auth_method 0 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.559+0000 7fe84951f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.559+0000 7fe84951f640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.559+0000 7fe84951f640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.560+0000 7fe848d1e640 10 monclient(hunting): get_auth_request con 0x7fe8440add90 auth_method 0 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.560+0000 7fe848d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.560+0000 7fe848d1e640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.560+0000 7fe848d1e640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.561+0000 7fe84951f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.561+0000 7fe84951f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.561+0000 7fe84951f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.561+0000 7fe848d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.561+0000 7fe848d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.561+0000 7fe848d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 10 monclient(hunting): handle_auth_done global_id 14817 payload 824 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 1 -- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe8441448e0 msgr2=0x7fe8440a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe8441448e0 0x7fe8440a7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 1 -- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8440add90 msgr2=0x7fe8441d5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:48.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8440add90 0x7fe8441d5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:48.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 10 monclient: _finish_hunting 0 2023-08-22T23:44:48.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 1 monclient: found mon.noname-a 2023-08-22T23:44:48.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:48.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:48.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.564+0000 7fe83bfff640 1 -- 172.21.15.29:0/3458509936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840009400 con 0x7fe84413e190 2023-08-22T23:44:48.571 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:33.853 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.572+0000 7fe83bfff640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 0x7fe8441d6770 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fe83400e9b0 tx=0x7fe83400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.572+0000 7fe8397fa640 10 client.?.objecter ms_handle_connect 0x7fe84413e190 2023-08-22T23:44:48.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.573+0000 7fe8397fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:48.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe834009670 con 0x7fe84413e190 2023-08-22T23:44:48.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:48.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: dump: 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: _finish_auth 0 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: _check_auth_tickets 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83400c9a0 con 0x7fe84413e190 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83400cd50 con 0x7fe84413e190 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe8397fa640 10 monclient: dump: 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:48.584 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.585 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:48.586 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:48.586 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:48.586 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:48.587 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:48.587 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:48.588 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:48.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe83affd640 4 set_mon_vals no callback set 2023-08-22T23:44:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 5 monclient: authenticate success, global_id 14817 2023-08-22T23:44:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 10 monclient: _renew_subs 2023-08-22T23:44:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe844141f00 con 0x7fe84413e190 2023-08-22T23:44:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 10 client.14817.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 10 monclient: _renew_subs 2023-08-22T23:44:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84413f0d0 con 0x7fe84413e190 2023-08-22T23:44:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.580+0000 7fe84af82640 1 librados: init done 2023-08-22T23:44:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.581+0000 7fe84af82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.581+0000 7fe84af82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:48.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.581+0000 7fe84af82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:48.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.581+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8441d63c0 con 0x7fe84413e190 2023-08-22T23:44:48.593 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:33.867 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.848+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe834013470 con 0x7fe84413e190 2023-08-22T23:44:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.848+0000 7fe8397fa640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe83006ba30 0x7fe83006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.848+0000 7fe848d1e640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe83006ba30 0x7fe83006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.848+0000 7fe848d1e640 10 monclient: get_auth_request con 0x7fe83006ba30 auth_method 0 2023-08-22T23:44:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.849+0000 7fe848d1e640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe83006ba30 0x7fe83006def0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe840036000 tx=0x7fe84003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.849+0000 7fe8397fa640 10 client.14817.objecter ms_handle_connect 0x7fe83006ba30 2023-08-22T23:44:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fe83405c0a0 con 0x7fe84413e190 2023-08-22T23:44:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 10 client.14817.objecter ms_dispatch 0x7fe844000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 3 client.14817.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 3 client.14817.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 20 client.14817.objecter dump_active .. 0 homeless 2023-08-22T23:44:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fe8340546d0 con 0x7fe84413e190 2023-08-22T23:44:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:48.866+0000 7fe8397fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.054+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8440ae200 con 0x7fe83006ba30 2023-08-22T23:44:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.054+0000 7fe8397fa640 1 -- 172.21.15.29:0/3458509936 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe8400319f0 con 0x7fe83006ba30 2023-08-22T23:44:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 5 to deep-scrub 2023-08-22T23:44:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 10 librados: watch_flush enter 2023-08-22T23:44:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 10 librados: watch_flush exit 2023-08-22T23:44:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 20 client.14817.objecter shutdown clearing up homeless session... 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 10 client.14817.objecter successfully canceled tick 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe83006ba30 msgr2=0x7fe83006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe83006ba30 0x7fe83006def0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe840036000 tx=0x7fe84003d040 comp rx=0 tx=0).stop 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 10 monclient: shutdown 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 msgr2=0x7fe8441d6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.056+0000 7fe84af82640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 0x7fe8441d6770 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fe83400e9b0 tx=0x7fe83400ee70 comp rx=0 tx=0).stop 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 shutdown_connections 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe83006ba30 0x7fe83006def0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe8441448e0 0x7fe8440a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84413e190 0x7fe8441d6770 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 --2- 172.21.15.29:0/3458509936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8440add90 0x7fe8441d5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 >> 172.21.15.29:0/3458509936 conn(0x7fe844065ed0 msgr2=0x7fe8440b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 shutdown_connections 2023-08-22T23:44:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 -- 172.21.15.29:0/3458509936 wait complete. 2023-08-22T23:44:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.057+0000 7fe84af82640 1 librados: shutdown 2023-08-22T23:44:49.071 INFO:tasks.ceph:Scrubbing osd.6 2023-08-22T23:44:49.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:49.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.204+0000 7f0a9fd5c640 10 monclient: get_monmap_and_config 2023-08-22T23:44:49.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.204+0000 7f0a9fd5c640 10 monclient: build_initial_monmap 2023-08-22T23:44:49.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.204+0000 7f0a9fd5c640 10 monclient: monmap: 2023-08-22T23:44:49.206 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:49.206 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:49.205834+0000 2023-08-22T23:44:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:49.205834+0000 2023-08-22T23:44:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:49.209 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 1 Processor -- start 2023-08-22T23:44:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 1 -- start start 2023-08-22T23:44:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 10 monclient: init 2023-08-22T23:44:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-08-22T23:44:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0a980a7950 0x7f0a980affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 10 monclient(hunting): picked mon.noname-c con 0x7f0a980a7950 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a980b0520 0x7f0a98141100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.205+0000 7f0a9fd5c640 10 monclient(hunting): picked mon.noname-b con 0x7f0a980b0520 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a98141640 0x7f0a98144680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 10 monclient(hunting): picked mon.noname-a con 0x7f0a98141640 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980a8b70 con 0x7f0a98141640 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980a9930 con 0x7f0a980a7950 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980a9c10 con 0x7f0a980b0520 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9fd5c640 10 monclient(hunting): authenticate will time out at 2853.456518s 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.206+0000 7f0a9daf8640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0a980a7950 0x7f0a980affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0a980a7950 0x7f0a980affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:56342/0 (socket says 172.21.15.29:56342) 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 1 -- 172.21.15.29:0/512461183 learned_addr learned my addr 172.21.15.29:0/512461183 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): get_auth_request con 0x7f0a980a7950 auth_method 0 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.207+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_done global_id 19636 payload 274 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 1 -- 172.21.15.29:0/512461183 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a980b0520 msgr2=0x7f0a98141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 1 --2- 172.21.15.29:0/512461183 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a980b0520 0x7f0a98141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 1 -- 172.21.15.29:0/512461183 >> v2:172.21.15.29:3300/0 conn(0x7f0a98141640 msgr2=0x7f0a98144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 1 --2- 172.21.15.29:0/512461183 >> v2:172.21.15.29:3300/0 conn(0x7f0a98141640 0x7f0a98144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 10 monclient: _finish_hunting 0 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 1 monclient: found mon.noname-c 2023-08-22T23:44:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.209+0000 7f0a9daf8640 1 -- 172.21.15.29:0/512461183 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980a9ef0 con 0x7f0a980a7950 2023-08-22T23:44:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.211+0000 7f0a9daf8640 1 --2- 172.21.15.29:0/512461183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a980affe0 secure :-1 s=READY pgs=4633 cs=0 l=1 rev1=1 crypto rx=0x7f0a98066480 tx=0x7f0a9402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad1506934a73de38 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 1 -- 172.21.15.29:0/512461183 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94004230 con 0x7f0a980a7950 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: dump: 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: _finish_auth 0 2023-08-22T23:44:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: _check_auth_tickets 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 1 -- 172.21.15.29:0/512461183 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a94004390 con 0x7f0a980a7950 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 1 -- 172.21.15.29:0/512461183 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94003810 con 0x7f0a980a7950 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9fd5c640 5 monclient: authenticate success, global_id 19636 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9fd5c640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9fd5c640 4 set_mon_vals no callback set 2023-08-22T23:44:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.214+0000 7f0a9caf6640 10 monclient: dump: 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 10 monclient: shutdown 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/512461183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 msgr2=0x7f0a980affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/512461183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a980affe0 secure :-1 s=READY pgs=4633 cs=0 l=1 rev1=1 crypto rx=0x7f0a98066480 tx=0x7f0a9402b860 comp rx=0 tx=0).stop 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/512461183 shutdown_connections 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/512461183 >> v2:172.21.15.29:3300/0 conn(0x7f0a98141640 0x7f0a98144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/512461183 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a980b0520 0x7f0a98141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/512461183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a980affe0 unknown :-1 s=CLOSED pgs=4633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/512461183 >> 172.21.15.29:0/512461183 conn(0x7f0a98065f70 msgr2=0x7f0a980b6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/512461183 shutdown_connections 2023-08-22T23:44:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.215+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/512461183 wait complete. 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 10 monclient: build_initial_monmap 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 10 monclient: monmap: 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 1 librados: starting msgr at 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 1 librados: starting objecter 2023-08-22T23:44:49.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 1 Processor -- start 2023-08-22T23:44:49.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 1 -- start start 2023-08-22T23:44:49.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 1 librados: setting wanted keys 2023-08-22T23:44:49.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 1 librados: calling monclient init 2023-08-22T23:44:49.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.216+0000 7f0a9fd5c640 10 monclient: init 2023-08-22T23:44:49.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:49.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:44:49.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a981405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): picked mon.noname-b con 0x7f0a980a7950 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:49.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a980b0520 0x7f0a980aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): picked mon.noname-a con 0x7f0a980b0520 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a98141640 0x7f0a980abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): picked mon.noname-c con 0x7f0a98141640 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980ac650 con 0x7f0a980b0520 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980ac310 con 0x7f0a980a7950 2023-08-22T23:44:49.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981430a0 con 0x7f0a98141640 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.217+0000 7f0a9fd5c640 10 monclient(hunting): authenticate will time out at 2853.467981s 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a981405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a981405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.29:56348/0 (socket says 172.21.15.29:56348) 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 1 -- 172.21.15.29:0/1765034020 learned_addr learned my addr 172.21.15.29:0/1765034020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 10 monclient(hunting): get_auth_request con 0x7f0a980a7950 auth_method 0 2023-08-22T23:44:49.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.218+0000 7f0a9daf8640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.219+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.219+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.219+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.219+0000 7f0a9e2f9640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a98141640 0x7f0a980abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.221+0000 7f0a9e2f9640 10 monclient(hunting): get_auth_request con 0x7f0a98141640 auth_method 0 2023-08-22T23:44:49.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.221+0000 7f0a9e2f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.221+0000 7f0a9e2f9640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.221+0000 7f0a9e2f9640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 10 monclient(hunting): handle_auth_done global_id 19639 payload 824 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 1 -- 172.21.15.29:0/1765034020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a98141640 msgr2=0x7f0a980abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f0a98141640 0x7f0a980abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 1 -- 172.21.15.29:0/1765034020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a980b0520 msgr2=0x7f0a980aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a980b0520 0x7f0a980aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 10 monclient: _finish_hunting 0 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 1 monclient: found mon.noname-b 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.222+0000 7f0a9daf8640 1 -- 172.21.15.29:0/1765034020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a94009460 con 0x7f0a980a7950 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.224+0000 7f0a9daf8640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a981405a0 secure :-1 s=READY pgs=4634 cs=0 l=1 rev1=1 crypto rx=0x7f0a94035040 tx=0x7f0a940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.224+0000 7f0a8e7fc640 10 client.?.objecter ms_handle_connect 0x7f0a980a7950 2023-08-22T23:44:49.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.224+0000 7f0a8e7fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:49.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94003e50 con 0x7f0a980a7950 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: dump: 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.235 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:49.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a94004500 con 0x7f0a980a7950 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94006240 con 0x7f0a980a7950 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:49.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8e7fc640 10 monclient: dump: 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.238 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 5 monclient: authenticate success, global_id 19639 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 10 monclient: _renew_subs 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a8ffff640 4 set_mon_vals no callback set 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a98143380 con 0x7f0a980a7950 2023-08-22T23:44:49.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 10 client.19639.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 10 monclient: _renew_subs 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980acce0 con 0x7f0a980a7950 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.229+0000 7f0a9fd5c640 1 librados: init done 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.230+0000 7f0a9fd5c640 10 client.19639.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.230+0000 7f0a9fd5c640 15 client.19639.objecter _session_command_op_assign -1 1 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.230+0000 7f0a9fd5c640 20 client.19639.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.230+0000 7f0a9fd5c640 10 client.19639.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:49.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.230+0000 7f0a9fd5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.230+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a98141ad0 con 0x7f0a980a7950 2023-08-22T23:44:49.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.251+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0a94039050 con 0x7f0a980a7950 2023-08-22T23:44:49.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.252+0000 7f0a8e7fc640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0a7c06b9e0 0x7f0a7c06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.252+0000 7f0a9d2f7640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0a7c06b9e0 0x7f0a7c06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.252+0000 7f0a9d2f7640 10 monclient: get_auth_request con 0x7f0a7c06b9e0 auth_method 0 2023-08-22T23:44:49.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.253+0000 7f0a8e7fc640 10 client.19639.objecter ms_handle_connect 0x7f0a7c06b9e0 2023-08-22T23:44:49.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.253+0000 7f0a9d2f7640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0a7c06b9e0 0x7f0a7c06dea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a980b5e00 tx=0x7f0a88006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f0a9407a740 con 0x7f0a980a7950 2023-08-22T23:44:49.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 10 client.19639.objecter ms_dispatch 0x7f0a98000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:49.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 3 client.19639.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:49.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 3 client.19639.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 10 client.19639.objecter checking command 1 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f0a7c0731e0 0x7f0a7c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 20 client.19639.objecter _get_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 15 client.19639.objecter _session_command_op_remove -1 1 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 20 client.19639.objecter _get_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 20 client.19639.objecter get_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 15 client.19639.objecter _session_command_op_assign 6 1 2023-08-22T23:44:49.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 4 2023-08-22T23:44:49.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 10 client.19639.objecter _send_command 1 2023-08-22T23:44:49.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a7c075d50 con 0x7f0a7c0731e0 2023-08-22T23:44:49.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 20 client.19639.objecter dump_active .. 0 homeless 2023-08-22T23:44:49.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a940763f0 con 0x7f0a980a7950 2023-08-22T23:44:49.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.254+0000 7f0a8e7fc640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:49.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.255+0000 7f0a9e2f9640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f0a7c0731e0 0x7f0a7c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.255+0000 7f0a9e2f9640 10 monclient: get_auth_request con 0x7f0a7c0731e0 auth_method 0 2023-08-22T23:44:49.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.258+0000 7f0a9e2f9640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f0a7c0731e0 0x7f0a7c075680 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.258+0000 7f0a8e7fc640 10 client.19639.objecter ms_handle_connect 0x7f0a7c0731e0 2023-08-22T23:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.282+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== osd.6 v2:172.21.15.52:6818/2556931765 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f0a7c075d50 con 0x7f0a7c0731e0 2023-08-22T23:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.282+0000 7f0a8e7fc640 10 client.19639.objecter ms_dispatch 0x7f0a98000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.282+0000 7f0a8e7fc640 10 client.19639.objecter _finish_command 1 = system:0 2023-08-22T23:44:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.282+0000 7f0a8e7fc640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 4 2023-08-22T23:44:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.282+0000 7f0a8e7fc640 15 client.19639.objecter _session_command_op_remove 6 1 2023-08-22T23:44:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 10 client.19639.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 15 client.19639.objecter _session_command_op_assign -1 2 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 20 client.19639.objecter _get_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 20 client.19639.objecter _get_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 15 client.19639.objecter _session_command_op_remove -1 2 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 20 client.19639.objecter get_session s=0x7f0a7c072b10 osd=6 3 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 15 client.19639.objecter _session_command_op_assign 6 2 2023-08-22T23:44:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 4 2023-08-22T23:44:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 10 client.19639.objecter _send_command 2 2023-08-22T23:44:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.295+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 --> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0a980a8600 con 0x7f0a7c0731e0 2023-08-22T23:44:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a8e7fc640 1 -- 172.21.15.29:0/1765034020 <== osd.6 v2:172.21.15.52:6818/2556931765 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0a90009b30 con 0x7f0a7c0731e0 2023-08-22T23:44:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a8e7fc640 10 client.19639.objecter ms_dispatch 0x7f0a98000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a8e7fc640 10 client.19639.objecter _finish_command 2 = system:0 2023-08-22T23:44:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a8e7fc640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 4 2023-08-22T23:44:49.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a8e7fc640 15 client.19639.objecter _session_command_op_remove 6 2 2023-08-22T23:44:49.312 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:49.312 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:49.312 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:49.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 10 librados: watch_flush enter 2023-08-22T23:44:49.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 10 librados: watch_flush exit 2023-08-22T23:44:49.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 10 client.19639.objecter close_session for osd.6 2023-08-22T23:44:49.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f0a7c0731e0 msgr2=0x7f0a7c075680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6818/2556931765,v1:172.21.15.52:6819/2556931765] conn(0x7f0a7c0731e0 0x7f0a7c075680 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 20 client.19639.objecter put_session s=0x7f0a7c072b10 osd=6 1 2023-08-22T23:44:49.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 20 client.19639.objecter shutdown clearing up homeless session... 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 10 client.19639.objecter successfully canceled tick 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0a7c06b9e0 msgr2=0x7f0a7c06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f0a7c06b9e0 0x7f0a7c06dea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a980b5e00 tx=0x7f0a88006c90 comp rx=0 tx=0).stop 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 10 monclient: shutdown 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 msgr2=0x7f0a981405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 --2- 172.21.15.29:0/1765034020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a980a7950 0x7f0a981405a0 secure :-1 s=READY pgs=4634 cs=0 l=1 rev1=1 crypto rx=0x7f0a94035040 tx=0x7f0a940046c0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9e2f9640 1 -- 172.21.15.29:0/1765034020 reap_dead start 2023-08-22T23:44:49.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 shutdown_connections 2023-08-22T23:44:49.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 >> 172.21.15.29:0/1765034020 conn(0x7f0a98065f70 msgr2=0x7f0a9813c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:49.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 shutdown_connections 2023-08-22T23:44:49.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 -- 172.21.15.29:0/1765034020 wait complete. 2023-08-22T23:44:49.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.310+0000 7f0a9fd5c640 1 librados: shutdown 2023-08-22T23:44:49.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-08-22T23:44:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.435+0000 7fed78209640 10 monclient: get_monmap_and_config 2023-08-22T23:44:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.435+0000 7fed78209640 10 monclient: build_initial_monmap 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.435+0000 7fed78209640 10 monclient: monmap: 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:49.437150+0000 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:49.437150+0000 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:49.438 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.436+0000 7fed78209640 1 Processor -- start 2023-08-22T23:44:49.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.436+0000 7fed78209640 1 -- start start 2023-08-22T23:44:49.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.436+0000 7fed78209640 10 monclient: init 2023-08-22T23:44:49.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:49.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-08-22T23:44:49.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed700a7d50 0x7fed700a8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): picked mon.noname-a con 0x7fed700a7d50 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:49.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fed700aff90 0x7fed701411e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): picked mon.noname-c con 0x7fed700aff90 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 0x7fed70141720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): picked mon.noname-b con 0x7fed700b03e0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700a87c0 con 0x7fed700a7d50 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700a9bf0 con 0x7fed700aff90 2023-08-22T23:44:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7013ee50 con 0x7fed700b03e0 2023-08-22T23:44:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed78209640 10 monclient(hunting): authenticate will time out at 2853.687908s 2023-08-22T23:44:49.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.437+0000 7fed767a6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 0x7fed70141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 0x7fed70141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34664/0 (socket says 172.21.15.29:34664) 2023-08-22T23:44:49.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 1 -- 172.21.15.29:0/2675579864 learned_addr learned my addr 172.21.15.29:0/2675579864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:49.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): get_auth_request con 0x7fed700b03e0 auth_method 0 2023-08-22T23:44:49.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed757a4640 1 --2- 172.21.15.29:0/2675579864 >> v2:172.21.15.52:3300/0 conn(0x7fed700aff90 0x7fed701411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed757a4640 10 monclient(hunting): get_auth_request con 0x7fed700aff90 auth_method 0 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed757a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed757a4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed757a4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.438+0000 7fed767a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.439+0000 7fed757a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.439+0000 7fed757a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.439+0000 7fed757a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 10 monclient(hunting): handle_auth_done global_id 16727 payload 274 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed75fa5640 1 --2- 172.21.15.29:0/2675579864 >> v2:172.21.15.29:3300/0 conn(0x7fed700a7d50 0x7fed700a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 1 -- 172.21.15.29:0/2675579864 >> v2:172.21.15.52:3300/0 conn(0x7fed700aff90 msgr2=0x7fed701411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 1 --2- 172.21.15.29:0/2675579864 >> v2:172.21.15.52:3300/0 conn(0x7fed700aff90 0x7fed701411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 1 -- 172.21.15.29:0/2675579864 >> v2:172.21.15.29:3300/0 conn(0x7fed700a7d50 msgr2=0x7fed700a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 1 --2- 172.21.15.29:0/2675579864 >> v2:172.21.15.29:3300/0 conn(0x7fed700a7d50 0x7fed700a8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 10 monclient: _finish_hunting 0 2023-08-22T23:44:49.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 1 monclient: found mon.noname-b 2023-08-22T23:44:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.440+0000 7fed767a6640 1 -- 172.21.15.29:0/2675579864 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70141c60 con 0x7fed700b03e0 2023-08-22T23:44:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.443+0000 7fed767a6640 1 --2- 172.21.15.29:0/2675579864 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 0x7fed70141720 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fed700b7a50 tx=0x7fed6802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d6011c087110e6 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 1 -- 172.21.15.29:0/2675579864 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6802c9e0 con 0x7fed700b03e0 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: dump: 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: _finish_auth 0 2023-08-22T23:44:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: _check_auth_tickets 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 1 -- 172.21.15.29:0/2675579864 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6802cb40 con 0x7fed700b03e0 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 1 -- 172.21.15.29:0/2675579864 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680343f0 con 0x7fed700b03e0 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 5 monclient: authenticate success, global_id 16727 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 4 set_mon_vals no callback set 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed74fa3640 10 monclient: dump: 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 10 monclient: shutdown 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:49.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 -- 172.21.15.29:0/2675579864 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 msgr2=0x7fed70141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 --2- 172.21.15.29:0/2675579864 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 0x7fed70141720 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fed700b7a50 tx=0x7fed6802be40 comp rx=0 tx=0).stop 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 -- 172.21.15.29:0/2675579864 shutdown_connections 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 --2- 172.21.15.29:0/2675579864 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700b03e0 0x7fed70141720 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 --2- 172.21.15.29:0/2675579864 >> v2:172.21.15.52:3300/0 conn(0x7fed700aff90 0x7fed701411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 --2- 172.21.15.29:0/2675579864 >> v2:172.21.15.29:3300/0 conn(0x7fed700a7d50 0x7fed700a8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 -- 172.21.15.29:0/2675579864 >> 172.21.15.29:0/2675579864 conn(0x7fed70065dd0 msgr2=0x7fed700b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 -- 172.21.15.29:0/2675579864 shutdown_connections 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.447+0000 7fed78209640 1 -- 172.21.15.29:0/2675579864 wait complete. 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.448+0000 7fed78209640 10 monclient: build_initial_monmap 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.448+0000 7fed78209640 10 monclient: monmap: 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.448+0000 7fed78209640 1 librados: starting msgr at 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.448+0000 7fed78209640 1 librados: starting objecter 2023-08-22T23:44:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 Processor -- start 2023-08-22T23:44:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 -- start start 2023-08-22T23:44:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 librados: setting wanted keys 2023-08-22T23:44:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 librados: calling monclient init 2023-08-22T23:44:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient: init 2023-08-22T23:44:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient: _add_conns ranks=[1,2,0] 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed700a7d50 0x7fed700b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): picked mon.noname-b con 0x7fed700a7d50 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700aff90 0x7fed701395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): picked mon.noname-c con 0x7fed700aff90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed700b03e0 0x7fed701d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): picked mon.noname-a con 0x7fed700b03e0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70135a40 con 0x7fed700b03e0 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed701cac40 con 0x7fed700a7d50 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed701caee0 con 0x7fed700aff90 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.449+0000 7fed78209640 10 monclient(hunting): authenticate will time out at 2853.700128s 2023-08-22T23:44:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700aff90 0x7fed701395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700aff90 0x7fed701395b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:34678/0 (socket says 172.21.15.29:34678) 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 1 -- 172.21.15.29:0/1947823358 learned_addr learned my addr 172.21.15.29:0/1947823358 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): get_auth_request con 0x7fed700aff90 auth_method 0 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed75fa5640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed700a7d50 0x7fed700b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed75fa5640 10 monclient(hunting): get_auth_request con 0x7fed700a7d50 auth_method 0 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed75fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed75fa5640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed75fa5640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.450+0000 7fed757a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:49.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.451+0000 7fed75fa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:49.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.451+0000 7fed75fa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:49.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.451+0000 7fed75fa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.451+0000 7fed767a6640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed700b03e0 0x7fed701d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.452+0000 7fed767a6640 10 monclient(hunting): get_auth_request con 0x7fed700b03e0 auth_method 0 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.452+0000 7fed767a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.452+0000 7fed767a6640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.452+0000 7fed767a6640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 10 monclient(hunting): handle_auth_done global_id 19645 payload 824 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 1 -- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700aff90 msgr2=0x7fed701395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700aff90 0x7fed701395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 1 -- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed700b03e0 msgr2=0x7fed701d5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed700b03e0 0x7fed701d5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 10 monclient: _finish_hunting 0 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 1 monclient: found mon.noname-b 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.453+0000 7fed75fa5640 1 -- 172.21.15.29:0/1947823358 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68009b30 con 0x7fed700a7d50 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.456+0000 7fed75fa5640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed700a7d50 0x7fed700b3590 secure :-1 s=READY pgs=4635 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c610 tx=0x7fed6c00cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.456+0000 7fed667fc640 10 client.?.objecter ms_handle_connect 0x7fed700a7d50 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.456+0000 7fed667fc640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c004220 con 0x7fed700a7d50 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: dump: 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c004380 con 0x7fed700a7d50 2023-08-22T23:44:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c003850 con 0x7fed700a7d50 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 5 monclient: authenticate success, global_id 19645 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 10 monclient: _renew_subs 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed700b3df0 con 0x7fed700a7d50 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed67fff640 4 set_mon_vals no callback set 2023-08-22T23:44:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed667fc640 10 monclient: dump: 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:49.473 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 10 client.19645.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 10 monclient: _renew_subs 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701d6450 con 0x7fed700a7d50 2023-08-22T23:44:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.459+0000 7fed78209640 1 librados: init done 2023-08-22T23:44:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.460+0000 7fed78209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.460+0000 7fed78209640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.460+0000 7fed78209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2023-08-22T23:44:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.460+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed700a9320 con 0x7fed700a7d50 2023-08-22T23:44:49.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.480+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fed6c0039b0 con 0x7fed700a7d50 2023-08-22T23:44:49.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.481+0000 7fed667fc640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed5406ba30 0x7fed5406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:49.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.481+0000 7fed757a4640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed5406ba30 0x7fed5406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:49.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.481+0000 7fed757a4640 10 monclient: get_auth_request con 0x7fed5406ba30 auth_method 0 2023-08-22T23:44:49.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed757a4640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed5406ba30 0x7fed5406def0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fed700ae2f0 tx=0x7fed60009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:49.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed667fc640 10 client.19645.objecter ms_handle_connect 0x7fed5406ba30 2023-08-22T23:44:49.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fed6c05b800 con 0x7fed700a7d50 2023-08-22T23:44:49.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed667fc640 10 client.19645.objecter ms_dispatch 0x7fed70000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:49.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed667fc640 3 client.19645.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:49.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed667fc640 3 client.19645.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:49.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.482+0000 7fed667fc640 20 client.19645.objecter dump_active .. 0 homeless 2023-08-22T23:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.737+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fed6c053e30 con 0x7fed700a7d50 2023-08-22T23:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.737+0000 7fed667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.737+0000 7fed667fc640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:49.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.916+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fed700a8d00 con 0x7fed5406ba30 2023-08-22T23:44:49.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.916+0000 7fed667fc640 1 -- 172.21.15.29:0/1947823358 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fed60011a90 con 0x7fed5406ba30 2023-08-22T23:44:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 6 to deep-scrub 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 10 librados: watch_flush enter 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 10 librados: watch_flush exit 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 20 client.19645.objecter shutdown clearing up homeless session... 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 10 client.19645.objecter successfully canceled tick 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed5406ba30 msgr2=0x7fed5406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed5406ba30 0x7fed5406def0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fed700ae2f0 tx=0x7fed60009190 comp rx=0 tx=0).stop 2023-08-22T23:44:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 10 monclient: shutdown 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed700a7d50 msgr2=0x7fed700b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed700a7d50 0x7fed700b3590 secure :-1 s=READY pgs=4635 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c610 tx=0x7fed6c00cad0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 shutdown_connections 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fed5406ba30 0x7fed5406def0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed700b03e0 0x7fed701d5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fed700aff90 0x7fed701395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 --2- 172.21.15.29:0/1947823358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed700a7d50 0x7fed700b3590 unknown :-1 s=CLOSED pgs=4635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 >> 172.21.15.29:0/1947823358 conn(0x7fed70065dd0 msgr2=0x7fed701cbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 shutdown_connections 2023-08-22T23:44:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 -- 172.21.15.29:0/1947823358 wait complete. 2023-08-22T23:44:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:49.918+0000 7fed78209640 1 librados: shutdown 2023-08-22T23:44:49.931 INFO:tasks.ceph:Scrubbing osd.7 2023-08-22T23:44:49.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-08-22T23:44:50.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.044+0000 7fd509ac5640 10 monclient: get_monmap_and_config 2023-08-22T23:44:50.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.044+0000 7fd509ac5640 10 monclient: build_initial_monmap 2023-08-22T23:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.044+0000 7fd509ac5640 10 monclient: monmap: 2023-08-22T23:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:50.045774+0000 2023-08-22T23:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:50.045774+0000 2023-08-22T23:44:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:50.047 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 1 Processor -- start 2023-08-22T23:44:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 1 -- start start 2023-08-22T23:44:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 10 monclient: init 2023-08-22T23:44:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5040a81b0 0x7fd5040a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 10 monclient(hunting): picked mon.noname-b con 0x7fd5040a81b0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd50413e760 0x7fd504144720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 10 monclient(hunting): picked mon.noname-a con 0x7fd50413e760 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.045+0000 7fd509ac5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd50413eba0 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5040b07e0 con 0x7fd50413eba0 2023-08-22T23:44:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5040a7d70 con 0x7fd50413e760 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040a98a0 con 0x7fd5040a81b0 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd509ac5640 10 monclient(hunting): authenticate will time out at 2854.296506s 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd5037fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5040a81b0 0x7fd5040a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd503fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fd50413eba0 0x7fd504144c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:57348/0 (socket says 172.21.15.29:57348) 2023-08-22T23:44:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.046+0000 7fd503fff640 1 -- 172.21.15.29:0/67603026 learned_addr learned my addr 172.21.15.29:0/67603026 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): get_auth_request con 0x7fd5040a81b0 auth_method 0 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.047+0000 7fd5037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 10 monclient(hunting): handle_auth_done global_id 16733 payload 274 2023-08-22T23:44:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 1 -- 172.21.15.29:0/67603026 >> v2:172.21.15.29:3300/0 conn(0x7fd50413e760 msgr2=0x7fd504144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 1 --2- 172.21.15.29:0/67603026 >> v2:172.21.15.29:3300/0 conn(0x7fd50413e760 0x7fd504144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 1 -- 172.21.15.29:0/67603026 >> v1:172.21.15.52:6789/0 conn(0x7fd50413eba0 legacy=0x7fd504144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 1 monclient: found mon.noname-b 2023-08-22T23:44:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:44:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.049+0000 7fd5037fe640 1 -- 172.21.15.29:0/67603026 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5040a8ad0 con 0x7fd5040a81b0 2023-08-22T23:44:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.052+0000 7fd5037fe640 1 --2- 172.21.15.29:0/67603026 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5040a81b0 0x7fd5040a8590 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40097a0 tx=0x7fd4f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f7cd5b0fb70c0b5 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 1 -- 172.21.15.29:0/67603026 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4031830 con 0x7fd5040a81b0 2023-08-22T23:44:50.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:44:50.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: dump: 2023-08-22T23:44:50.118 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.118 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 1 -- 172.21.15.29:0/67603026 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f4031990 con 0x7fd5040a81b0 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:50.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd509ac5640 5 monclient: authenticate success, global_id 16733 2023-08-22T23:44:50.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 1 -- 172.21.15.29:0/67603026 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4031d20 con 0x7fd5040a81b0 2023-08-22T23:44:50.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd509ac5640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:50.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd509ac5640 4 set_mon_vals no callback set 2023-08-22T23:44:50.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:44:50.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.115+0000 7fd5027fc640 10 monclient: dump: 2023-08-22T23:44:50.122 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.122 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.125 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.126 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 10 monclient: shutdown 2023-08-22T23:44:50.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 -- 172.21.15.29:0/67603026 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5040a81b0 msgr2=0x7fd5040a8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 --2- 172.21.15.29:0/67603026 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5040a81b0 0x7fd5040a8590 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40097a0 tx=0x7fd4f402b8d0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 -- 172.21.15.29:0/67603026 shutdown_connections 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 --2- 172.21.15.29:0/67603026 >> v2:172.21.15.29:3300/0 conn(0x7fd50413e760 0x7fd504144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 --2- 172.21.15.29:0/67603026 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5040a81b0 0x7fd5040a8590 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 -- 172.21.15.29:0/67603026 >> 172.21.15.29:0/67603026 conn(0x7fd504065fb0 msgr2=0x7fd5040b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 -- 172.21.15.29:0/67603026 shutdown_connections 2023-08-22T23:44:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.116+0000 7fd509ac5640 1 -- 172.21.15.29:0/67603026 wait complete. 2023-08-22T23:44:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.117+0000 7fd509ac5640 10 monclient: build_initial_monmap 2023-08-22T23:44:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.117+0000 7fd509ac5640 10 monclient: monmap: 2023-08-22T23:44:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:50.133 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:50.133 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:50.133 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.117+0000 7fd509ac5640 1 librados: starting msgr at 2023-08-22T23:44:50.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.117+0000 7fd509ac5640 1 librados: starting objecter 2023-08-22T23:44:50.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.117+0000 7fd509ac5640 1 Processor -- start 2023-08-22T23:44:50.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 -- start start 2023-08-22T23:44:50.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 librados: setting wanted keys 2023-08-22T23:44:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 librados: calling monclient init 2023-08-22T23:44:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient: init 2023-08-22T23:44:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient: _add_conns ranks=[1,0,2] 2023-08-22T23:44:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5040a81b0 0x7fd5041430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): picked mon.noname-b con 0x7fd5040a81b0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50413e760 0x7fd5040ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): picked mon.noname-a con 0x7fd50413e760 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5041d5e60 0x7fd5041d68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): picked mon.noname-c con 0x7fd5041d5e60 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd504140090 con 0x7fd50413e760 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5041d6de0 con 0x7fd5040a81b0 2023-08-22T23:44:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5041d6380 con 0x7fd5041d5e60 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.118+0000 7fd509ac5640 10 monclient(hunting): authenticate will time out at 2854.369145s 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50413e760 0x7fd5040ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50413e760 0x7fd5040ac1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48116/0 (socket says 172.21.15.29:48116) 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 1 -- 172.21.15.29:0/3192199840 learned_addr learned my addr 172.21.15.29:0/3192199840 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 10 monclient(hunting): get_auth_request con 0x7fd50413e760 auth_method 0 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.119+0000 7fd502ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.120+0000 7fd502ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.120+0000 7fd502ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.120+0000 7fd502ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd503fff640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5041d5e60 0x7fd5041d68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 10 monclient(hunting): handle_auth_done global_id 14823 payload 824 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 1 -- 172.21.15.29:0/3192199840 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5041d5e60 msgr2=0x7fd5041d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd5041d5e60 0x7fd5041d68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 1 -- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5040a81b0 msgr2=0x7fd5041430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5040a81b0 0x7fd5041430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:44:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 1 monclient: found mon.noname-a 2023-08-22T23:44:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.123+0000 7fd502ffd640 1 -- 172.21.15.29:0/3192199840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4009400 con 0x7fd50413e760 2023-08-22T23:44:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.126+0000 7fd502ffd640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50413e760 0x7fd5040ac1f0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800b4a0 tx=0x7fd4f800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.126+0000 7fd4ebfff640 10 client.?.objecter ms_handle_connect 0x7fd50413e760 2023-08-22T23:44:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.126+0000 7fd4ebfff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8010610 con 0x7fd50413e760 2023-08-22T23:44:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: dump: 2023-08-22T23:44:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: _finish_auth 0 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8010770 con 0x7fd50413e760 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 5 monclient: authenticate success, global_id 14823 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 10 monclient: _renew_subs 2023-08-22T23:44:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5040acda0 con 0x7fd50413e760 2023-08-22T23:44:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8010b20 con 0x7fd50413e760 2023-08-22T23:44:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd4ebfff640 10 monclient: dump: 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd501ffb640 4 set_mon_vals no callback set 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 10 client.14823.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 10 monclient: _renew_subs 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50413f170 con 0x7fd50413e760 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.130+0000 7fd509ac5640 1 librados: init done 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.131+0000 7fd509ac5640 10 client.14823.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.131+0000 7fd509ac5640 15 client.14823.objecter _session_command_op_assign -1 1 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.131+0000 7fd509ac5640 20 client.14823.objecter _get_session osd=-1 returning homeless 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.131+0000 7fd509ac5640 10 client.14823.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.131+0000 7fd509ac5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.131+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5040b0bc0 con 0x7fd50413e760 2023-08-22T23:44:50.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.154+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd4f8010c80 con 0x7fd50413e760 2023-08-22T23:44:50.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.154+0000 7fd4ebfff640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd4f006ba30 0x7fd4f006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.154+0000 7fd5037fe640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd4f006ba30 0x7fd4f006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.154+0000 7fd5037fe640 10 monclient: get_auth_request con 0x7fd4f006ba30 auth_method 0 2023-08-22T23:44:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.155+0000 7fd5037fe640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd4f006ba30 0x7fd4f006def0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4036000 tx=0x7fd4f4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.155+0000 7fd4ebfff640 10 client.14823.objecter ms_handle_connect 0x7fd4f006ba30 2023-08-22T23:44:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fd4f80581d0 con 0x7fd50413e760 2023-08-22T23:44:50.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 10 client.14823.objecter ms_dispatch 0x7fd504000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:50.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 3 client.14823.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:50.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 3 client.14823.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:50.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 10 client.14823.objecter checking command 1 2023-08-22T23:44:50.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7fd4f0073230 0x7fd4f00756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 20 client.14823.objecter _get_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 15 client.14823.objecter _session_command_op_remove -1 1 2023-08-22T23:44:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 20 client.14823.objecter _get_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 20 client.14823.objecter get_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 15 client.14823.objecter _session_command_op_assign 7 1 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 4 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 10 client.14823.objecter _send_command 1 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 --> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4f0075da0 con 0x7fd4f0073230 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 20 client.14823.objecter dump_active .. 0 homeless 2023-08-22T23:44:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=918) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4f8053e80 con 0x7fd50413e760 2023-08-22T23:44:50.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.157+0000 7fd4ebfff640 10 monclient: handle_get_version_reply finishing 1 version 918 2023-08-22T23:44:50.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.158+0000 7fd503fff640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7fd4f0073230 0x7fd4f00756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.158+0000 7fd503fff640 10 monclient: get_auth_request con 0x7fd4f0073230 auth_method 0 2023-08-22T23:44:50.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.161+0000 7fd4ebfff640 10 client.14823.objecter ms_handle_connect 0x7fd4f0073230 2023-08-22T23:44:50.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.161+0000 7fd503fff640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7fd4f0073230 0x7fd4f00756d0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.185+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== osd.7 v2:172.21.15.52:6810/3454580439 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fd4f0075da0 con 0x7fd4f0073230 2023-08-22T23:44:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.185+0000 7fd4ebfff640 10 client.14823.objecter ms_dispatch 0x7fd504000b90 command_reply(tid 1: 0 ) v1 2023-08-22T23:44:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.185+0000 7fd4ebfff640 10 client.14823.objecter _finish_command 1 = system:0 2023-08-22T23:44:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.185+0000 7fd4ebfff640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 4 2023-08-22T23:44:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.185+0000 7fd4ebfff640 15 client.14823.objecter _session_command_op_remove 7 1 2023-08-22T23:44:50.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 10 client.14823.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-08-22T23:44:50.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 15 client.14823.objecter _session_command_op_assign -1 2 2023-08-22T23:44:50.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 20 client.14823.objecter _get_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 20 client.14823.objecter _get_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 15 client.14823.objecter _session_command_op_remove -1 2 2023-08-22T23:44:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 20 client.14823.objecter get_session s=0x7fd4f0072b60 osd=7 3 2023-08-22T23:44:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 15 client.14823.objecter _session_command_op_assign 7 2 2023-08-22T23:44:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 4 2023-08-22T23:44:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 10 client.14823.objecter _send_command 2 2023-08-22T23:44:50.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.198+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 --> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd5040a9df0 con 0x7fd4f0073230 2023-08-22T23:44:50.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd4ebfff640 1 -- 172.21.15.29:0/3192199840 <== osd.7 v2:172.21.15.52:6810/3454580439 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd4fc001c20 con 0x7fd4f0073230 2023-08-22T23:44:50.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd4ebfff640 10 client.14823.objecter ms_dispatch 0x7fd504000b90 command_reply(tid 2: 0 ) v1 2023-08-22T23:44:50.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd4ebfff640 10 client.14823.objecter _finish_command 2 = system:0 2023-08-22T23:44:50.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd4ebfff640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 4 2023-08-22T23:44:50.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd4ebfff640 15 client.14823.objecter _session_command_op_remove 7 2 2023-08-22T23:44:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-08-22T23:44:50.215 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-08-22T23:44:50.216 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-08-22T23:44:50.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 10 librados: watch_flush enter 2023-08-22T23:44:50.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 10 librados: watch_flush exit 2023-08-22T23:44:50.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 10 client.14823.objecter close_session for osd.7 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7fd4f0073230 msgr2=0x7fd4f00756d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6810/3454580439,v1:172.21.15.52:6811/3454580439] conn(0x7fd4f0073230 0x7fd4f00756d0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 20 client.14823.objecter put_session s=0x7fd4f0072b60 osd=7 1 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 20 client.14823.objecter shutdown clearing up homeless session... 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 10 client.14823.objecter successfully canceled tick 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd4f006ba30 msgr2=0x7fd4f006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd4f006ba30 0x7fd4f006def0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4036000 tx=0x7fd4f4030000 comp rx=0 tx=0).stop 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 10 monclient: shutdown 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50413e760 msgr2=0x7fd5040ac1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 --2- 172.21.15.29:0/3192199840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd50413e760 0x7fd5040ac1f0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800b4a0 tx=0x7fd4f800b960 comp rx=0 tx=0).stop 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd503fff640 1 -- 172.21.15.29:0/3192199840 reap_dead start 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 shutdown_connections 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 >> 172.21.15.29:0/3192199840 conn(0x7fd504065fb0 msgr2=0x7fd5040a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 shutdown_connections 2023-08-22T23:44:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 -- 172.21.15.29:0/3192199840 wait complete. 2023-08-22T23:44:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.213+0000 7fd509ac5640 1 librados: shutdown 2023-08-22T23:44:50.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-08-22T23:44:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.337+0000 7f9dede95640 10 monclient: get_monmap_and_config 2023-08-22T23:44:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.337+0000 7f9dede95640 10 monclient: build_initial_monmap 2023-08-22T23:44:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.337+0000 7f9dede95640 10 monclient: monmap: 2023-08-22T23:44:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:50.339005+0000 2023-08-22T23:44:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:50.339005+0000 2023-08-22T23:44:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:50.342 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.338+0000 7f9dede95640 1 Processor -- start 2023-08-22T23:44:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.338+0000 7f9dede95640 1 -- start start 2023-08-22T23:44:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.338+0000 7f9dede95640 10 monclient: init 2023-08-22T23:44:50.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:50.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-08-22T23:44:50.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f9de806fd50 0x7f9de80af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient(hunting): picked mon.noname-c con 0x7f9de806fd50 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:50.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9de8070130 addr v1:172.21.15.52:6789/0 2023-08-22T23:44:50.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9de813c1d0 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:50.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80aebd0 con 0x7f9de813c1d0 2023-08-22T23:44:50.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80aee20 con 0x7f9de8070130 2023-08-22T23:44:50.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de8143430 con 0x7f9de806fd50 2023-08-22T23:44:50.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:50.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9dede95640 10 monclient(hunting): authenticate will time out at 2854.589871s 2023-08-22T23:44:50.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9de7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9de813c1d0 0x7f9de813c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59804/0 (socket says 172.21.15.29:59804) 2023-08-22T23:44:50.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.339+0000 7f9de7fff640 1 -- 172.21.15.29:0/1221796930 learned_addr learned my addr 172.21.15.29:0/1221796930 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.340+0000 7f9de77fe640 1 --2- 172.21.15.29:0/1221796930 >> v2:172.21.15.52:3300/0 conn(0x7f9de806fd50 0x7f9de80af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.340+0000 7f9de77fe640 10 monclient(hunting): get_auth_request con 0x7f9de806fd50 auth_method 0 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.340+0000 7f9de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.340+0000 7f9de77fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.340+0000 7f9de77fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.341+0000 7f9de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.341+0000 7f9de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.341+0000 7f9de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.342+0000 7f9de67fc640 1 -- 172.21.15.29:0/1221796930 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219627847 0 0) 0x7f9de80aebd0 con 0x7f9de813c1d0 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.342+0000 7f9de67fc640 10 monclient(hunting): _negotiate 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.342+0000 7f9de67fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.342+0000 7f9de67fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.342+0000 7f9de67fc640 10 monclient(hunting): my global_id is 14826 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.342+0000 7f9de67fc640 1 -- 172.21.15.29:0/1221796930 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd0003520 con 0x7f9de813c1d0 2023-08-22T23:44:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 10 monclient(hunting): handle_auth_done global_id 19648 payload 274 2023-08-22T23:44:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 1 -- 172.21.15.29:0/1221796930 >> v1:172.21.15.52:6789/0 conn(0x7f9de8070130 legacy=0x7f9de8142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 1 -- 172.21.15.29:0/1221796930 >> v1:172.21.15.29:6789/0 conn(0x7f9de813c1d0 legacy=0x7f9de813c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 1 monclient: found mon.noname-c 2023-08-22T23:44:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.343+0000 7f9de77fe640 1 -- 172.21.15.29:0/1221796930 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80a7cf0 con 0x7f9de806fd50 2023-08-22T23:44:50.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.346+0000 7f9de77fe640 1 --2- 172.21.15.29:0/1221796930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de806fd50 0x7f9de80af150 secure :-1 s=READY pgs=4638 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80097a0 tx=0x7f9dd802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b59e33b47453a89 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 1 -- 172.21.15.29:0/1221796930 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8030610 con 0x7f9de806fd50 2023-08-22T23:44:50.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:50.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: dump: 2023-08-22T23:44:50.357 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.358 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.359 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 1 -- 172.21.15.29:0/1221796930 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd8030770 con 0x7f9de806fd50 2023-08-22T23:44:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 1 -- 172.21.15.29:0/1221796930 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8030b00 con 0x7f9de806fd50 2023-08-22T23:44:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9de67fc640 10 monclient: dump: 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9dede95640 5 monclient: authenticate success, global_id 19648 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9dede95640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.350+0000 7f9dede95640 4 set_mon_vals no callback set 2023-08-22T23:44:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 10 monclient: shutdown 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 -- 172.21.15.29:0/1221796930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de806fd50 msgr2=0x7f9de80af150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 --2- 172.21.15.29:0/1221796930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de806fd50 0x7f9de80af150 secure :-1 s=READY pgs=4638 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80097a0 tx=0x7f9dd802b960 comp rx=0 tx=0).stop 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 -- 172.21.15.29:0/1221796930 shutdown_connections 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 --2- 172.21.15.29:0/1221796930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de806fd50 0x7f9de80af150 unknown :-1 s=CLOSED pgs=4638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 -- 172.21.15.29:0/1221796930 >> 172.21.15.29:0/1221796930 conn(0x7f9de80b47d0 msgr2=0x7f9de80b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 -- 172.21.15.29:0/1221796930 shutdown_connections 2023-08-22T23:44:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.351+0000 7f9dede95640 1 -- 172.21.15.29:0/1221796930 wait complete. 2023-08-22T23:44:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 10 monclient: build_initial_monmap 2023-08-22T23:44:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 10 monclient: monmap: 2023-08-22T23:44:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:50.367 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 1 librados: starting msgr at 2023-08-22T23:44:50.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 1 librados: starting objecter 2023-08-22T23:44:50.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 1 Processor -- start 2023-08-22T23:44:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 1 -- start start 2023-08-22T23:44:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 1 librados: setting wanted keys 2023-08-22T23:44:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 1 librados: calling monclient init 2023-08-22T23:44:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.352+0000 7f9dede95640 10 monclient: init 2023-08-22T23:44:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:44:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 0x7f9de8141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): picked mon.noname-a con 0x7f9de806fd50 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de8070130 0x7f9de81d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): picked mon.noname-b con 0x7f9de8070130 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9de813c1d0 0x7f9de81d72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): picked mon.noname-c con 0x7f9de813c1d0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:50.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de813f6e0 con 0x7f9de806fd50 2023-08-22T23:44:50.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de813eeb0 con 0x7f9de8070130 2023-08-22T23:44:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81d1880 con 0x7f9de813c1d0 2023-08-22T23:44:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9dede95640 10 monclient(hunting): authenticate will time out at 2854.603654s 2023-08-22T23:44:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 0x7f9de8141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de6ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de8070130 0x7f9de81d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9de813c1d0 0x7f9de81d72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 0x7f9de8141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48132/0 (socket says 172.21.15.29:48132) 2023-08-22T23:44:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 1 -- 172.21.15.29:0/2563981961 learned_addr learned my addr 172.21.15.29:0/2563981961 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 10 monclient(hunting): get_auth_request con 0x7f9de806fd50 auth_method 0 2023-08-22T23:44:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.353+0000 7f9de77fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): get_auth_request con 0x7f9de813c1d0 auth_method 0 2023-08-22T23:44:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): get_auth_request con 0x7f9de8070130 auth_method 0 2023-08-22T23:44:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.354+0000 7f9de6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 10 monclient(hunting): handle_auth_done global_id 14829 payload 824 2023-08-22T23:44:50.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 1 -- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9de813c1d0 msgr2=0x7f9de81d72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9de813c1d0 0x7f9de81d72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 1 -- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de8070130 msgr2=0x7f9de81d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de8070130 0x7f9de81d3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 10 monclient: _finish_hunting 0 2023-08-22T23:44:50.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 1 monclient: found mon.noname-a 2023-08-22T23:44:50.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:50.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.357+0000 7f9de77fe640 1 -- 172.21.15.29:0/2563981961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd8009400 con 0x7f9de806fd50 2023-08-22T23:44:50.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.360+0000 7f9de77fe640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 0x7f9de8141f40 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f9dd802bf70 tx=0x7f9dd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.360+0000 7f9dcffff640 10 client.?.objecter ms_handle_connect 0x7f9de806fd50 2023-08-22T23:44:50.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.360+0000 7f9dcffff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:50.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd803e040 con 0x7f9de806fd50 2023-08-22T23:44:50.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:50.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: dump: 2023-08-22T23:44:50.385 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.386 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.386 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.386 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.386 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.386 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.387 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.387 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.387 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.387 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: _finish_auth 0 2023-08-22T23:44:50.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd8039cb0 con 0x7f9de806fd50 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8044630 con 0x7f9de806fd50 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 5 monclient: authenticate success, global_id 14829 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 10 monclient: _renew_subs 2023-08-22T23:44:50.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de813e200 con 0x7f9de806fd50 2023-08-22T23:44:50.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:50.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dcffff640 10 monclient: dump: 2023-08-22T23:44:50.389 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.389 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.390 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.390 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.390 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.390 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.390 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.391 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.391 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.391 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9de5ffb640 4 set_mon_vals no callback set 2023-08-22T23:44:50.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 10 client.14829.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:50.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 10 monclient: _renew_subs 2023-08-22T23:44:50.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.363+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de813fd10 con 0x7f9de806fd50 2023-08-22T23:44:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.364+0000 7f9dede95640 1 librados: init done 2023-08-22T23:44:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.364+0000 7f9dede95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.364+0000 7f9dede95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.364+0000 7f9dede95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.364+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de80714d0 con 0x7f9de806fd50 2023-08-22T23:44:50.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.383+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9dd804d3b0 con 0x7f9de806fd50 2023-08-22T23:44:50.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.383+0000 7f9dcffff640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9dd406ba30 0x7f9dd406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.383+0000 7f9de6ffd640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9dd406ba30 0x7f9dd406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.383+0000 7f9de6ffd640 10 monclient: get_auth_request con 0x7f9dd406ba30 auth_method 0 2023-08-22T23:44:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.384+0000 7f9de6ffd640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9dd406ba30 0x7f9dd406def0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9de813c1a0 tx=0x7f9ddc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.384+0000 7f9dcffff640 10 client.14829.objecter ms_handle_connect 0x7f9dd406ba30 2023-08-22T23:44:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.385+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f9dd8039e20 con 0x7f9de806fd50 2023-08-22T23:44:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.385+0000 7f9dcffff640 10 client.14829.objecter ms_dispatch 0x7f9de8000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.385+0000 7f9dcffff640 3 client.14829.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.385+0000 7f9dcffff640 3 client.14829.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.385+0000 7f9dcffff640 20 client.14829.objecter dump_active .. 0 homeless 2023-08-22T23:44:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.633+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f9dd80766a0 con 0x7f9de806fd50 2023-08-22T23:44:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.633+0000 7f9dcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.633+0000 7f9dcffff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:50.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.813+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9de8071070 con 0x7f9dd406ba30 2023-08-22T23:44:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.814+0000 7f9dcffff640 1 -- 172.21.15.29:0/2563981961 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9ddc0014c0 con 0x7f9dd406ba30 2023-08-22T23:44:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 7 to deep-scrub 2023-08-22T23:44:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 10 librados: watch_flush enter 2023-08-22T23:44:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 10 librados: watch_flush exit 2023-08-22T23:44:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 20 client.14829.objecter shutdown clearing up homeless session... 2023-08-22T23:44:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 10 client.14829.objecter successfully canceled tick 2023-08-22T23:44:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9dd406ba30 msgr2=0x7f9dd406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9dd406ba30 0x7f9dd406def0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9de813c1a0 tx=0x7f9ddc009040 comp rx=0 tx=0).stop 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 10 monclient: shutdown 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.815+0000 7f9dede95640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 msgr2=0x7f9de8141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 0x7f9de8141f40 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f9dd802bf70 tx=0x7f9dd8004730 comp rx=0 tx=0).stop 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 shutdown_connections 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f9dd406ba30 0x7f9dd406def0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f9de813c1d0 0x7f9de81d72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9de8070130 0x7f9de81d3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 --2- 172.21.15.29:0/2563981961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de806fd50 0x7f9de8141f40 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 >> 172.21.15.29:0/2563981961 conn(0x7f9de80b47d0 msgr2=0x7f9de8132810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 shutdown_connections 2023-08-22T23:44:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 -- 172.21.15.29:0/2563981961 wait complete. 2023-08-22T23:44:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.816+0000 7f9dede95640 1 librados: shutdown 2023-08-22T23:44:50.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:44:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.934+0000 7f3a5933d640 10 monclient: get_monmap_and_config 2023-08-22T23:44:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.934+0000 7f3a5933d640 10 monclient: build_initial_monmap 2023-08-22T23:44:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.934+0000 7f3a5933d640 10 monclient: monmap: 2023-08-22T23:44:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:44:50.935783+0000 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:44:50.935783+0000 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:44:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:44:50.938 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 Processor -- start 2023-08-22T23:44:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 -- start start 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient: init 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a5413e860 addr v1:172.21.15.29:6789/0 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f3a54144710 0x7f3a54144b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a54144710 addr v2:172.21.15.52:3300/0 2023-08-22T23:44:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a54149a90 0x7f3a5414be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a54149a90 addr v2:172.21.15.29:3300/0 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540a7ce0 con 0x7f3a5413e860 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a540a7f80 con 0x7f3a54149a90 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a540a95a0 con 0x7f3a54144710 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.935+0000 7f3a5933d640 10 monclient(hunting): authenticate will time out at 2855.186352s 2023-08-22T23:44:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a537fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a54149a90 0x7f3a5414be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a4a7fc640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f3a54144710 0x7f3a54144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a52ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3a5413e860 0x7f3a540a74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59806/0 (socket says 172.21.15.29:59806) 2023-08-22T23:44:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a52ffd640 1 -- 172.21.15.29:0/2116566147 learned_addr learned my addr 172.21.15.29:0/2116566147 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:44:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a4a7fc640 10 monclient(hunting): get_auth_request con 0x7f3a54144710 auth_method 0 2023-08-22T23:44:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.936+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a537fe640 10 monclient(hunting): get_auth_request con 0x7f3a54149a90 auth_method 0 2023-08-22T23:44:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a537fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a537fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.937+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.938+0000 7f3a537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.938+0000 7f3a537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.938+0000 7f3a537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_done global_id 19654 payload 274 2023-08-22T23:44:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/2116566147 >> v2:172.21.15.29:3300/0 conn(0x7f3a54149a90 msgr2=0x7f3a5414be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:44:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 1 --2- 172.21.15.29:0/2116566147 >> v2:172.21.15.29:3300/0 conn(0x7f3a54149a90 0x7f3a5414be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/2116566147 >> v1:172.21.15.29:6789/0 conn(0x7f3a5413e860 legacy=0x7f3a540a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 10 monclient: _finish_hunting 0 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 1 monclient: found mon.noname-c 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.939+0000 7f3a4a7fc640 1 -- 172.21.15.29:0/2116566147 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540a9880 con 0x7f3a54144710 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.941+0000 7f3a4a7fc640 1 --2- 172.21.15.29:0/2116566147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a54144b30 secure :-1 s=READY pgs=4639 cs=0 l=1 rev1=1 crypto rx=0x7f3a540b7850 tx=0x7f3a3c02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4e3a022c4c0f9db server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 1 -- 172.21.15.29:0/2116566147 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c031710 con 0x7f3a54144710 2023-08-22T23:44:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: dump: 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.951 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: _finish_auth 0 2023-08-22T23:44:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: _check_auth_tickets 2023-08-22T23:44:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 5 monclient: authenticate success, global_id 19654 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 1 -- 172.21.15.29:0/2116566147 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c02f050 con 0x7f3a54144710 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 10 monclient: get_monmap_and_config success 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 4 set_mon_vals no callback set 2023-08-22T23:44:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 1 -- 172.21.15.29:0/2116566147 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c031a90 con 0x7f3a54144710 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a527fc640 10 monclient: dump: 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.955 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 10 monclient: shutdown 2023-08-22T23:44:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 -- 172.21.15.29:0/2116566147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 msgr2=0x7f3a54144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 --2- 172.21.15.29:0/2116566147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a54144b30 secure :-1 s=READY pgs=4639 cs=0 l=1 rev1=1 crypto rx=0x7f3a540b7850 tx=0x7f3a3c02b7b0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 -- 172.21.15.29:0/2116566147 shutdown_connections 2023-08-22T23:44:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 --2- 172.21.15.29:0/2116566147 >> v2:172.21.15.29:3300/0 conn(0x7f3a54149a90 0x7f3a5414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 --2- 172.21.15.29:0/2116566147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a54144b30 unknown :-1 s=CLOSED pgs=4639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 -- 172.21.15.29:0/2116566147 >> 172.21.15.29:0/2116566147 conn(0x7f3a54065f10 msgr2=0x7f3a540b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 -- 172.21.15.29:0/2116566147 shutdown_connections 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.945+0000 7f3a5933d640 1 -- 172.21.15.29:0/2116566147 wait complete. 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.946+0000 7f3a5933d640 10 monclient: build_initial_monmap 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.946+0000 7f3a5933d640 10 monclient: monmap: 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:44:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:44:50.960 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.946+0000 7f3a5933d640 1 librados: starting msgr at 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.946+0000 7f3a5933d640 1 librados: starting objecter 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.946+0000 7f3a5933d640 1 Processor -- start 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 -- start start 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 librados: setting wanted keys 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 librados: calling monclient init 2023-08-22T23:44:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient: init 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient: _reopen_session rank -1 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 0x7f3a541d95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a5413e860 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a5413fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a54144710 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:44:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a54149a90 0x7f3a541402d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a54149a90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54149690 con 0x7f3a5413e860 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541da0f0 con 0x7f3a54144710 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): start opening mon connection 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54140810 con 0x7f3a54149a90 2023-08-22T23:44:50.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): _renew_subs 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.947+0000 7f3a5933d640 10 monclient(hunting): authenticate will time out at 2855.198119s 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 0x7f3a541d95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a537fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a54149a90 0x7f3a541402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 0x7f3a541d95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48144/0 (socket says 172.21.15.29:48144) 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 1 -- 172.21.15.29:0/3738137324 learned_addr learned my addr 172.21.15.29:0/3738137324 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): get_auth_request con 0x7f3a5413e860 auth_method 0 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a537fe640 10 monclient(hunting): get_auth_request con 0x7f3a54149a90 auth_method 0 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a537fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a537fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a4a7fc640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a5413fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a4a7fc640 10 monclient(hunting): get_auth_request con 0x7f3a54144710 auth_method 0 2023-08-22T23:44:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:44:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:44:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:44:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.948+0000 7f3a52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.949+0000 7f3a537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.949+0000 7f3a537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.949+0000 7f3a537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.949+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.949+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.949+0000 7f3a4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 10 monclient(hunting): handle_auth_done global_id 14835 payload 824 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 1 -- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a54149a90 msgr2=0x7f3a541402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a54149a90 0x7f3a541402d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 1 -- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 msgr2=0x7f3a5413fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a5413fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 10 monclient: _finish_hunting 0 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 1 monclient: found mon.noname-a 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.951+0000 7f3a52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.952+0000 7f3a52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.952+0000 7f3a52ffd640 1 -- 172.21.15.29:0/3738137324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c009400 con 0x7f3a5413e860 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.954+0000 7f3a52ffd640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 0x7f3a541d95e0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c480 tx=0x7f3a4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.954+0000 7f3a4bfff640 10 client.?.objecter ms_handle_connect 0x7f3a5413e860 2023-08-22T23:44:50.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.954+0000 7f3a4bfff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:44:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44007480 con 0x7f3a5413e860 2023-08-22T23:44:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:44:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 10 monclient: dump: 2023-08-22T23:44:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 10 monclient: _finish_auth 0 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 10 monclient: _check_auth_tickets 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a440075e0 con 0x7f3a5413e860 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a5933d640 5 monclient: authenticate success, global_id 14835 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a5933d640 10 monclient: _renew_subs 2023-08-22T23:44:50.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a5933d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.957+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a54148e40 con 0x7f3a5413e860 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a4bfff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440059a0 con 0x7f3a5413e860 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a4bfff640 10 monclient: dump: 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:44:50.974 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:44:50.975 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:44:50.976 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:44:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a51ffb640 4 set_mon_vals no callback set 2023-08-22T23:44:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 10 client.14835.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:44:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 10 monclient: _renew_subs 2023-08-22T23:44:50.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a541d9d00 con 0x7f3a5413e860 2023-08-22T23:44:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 1 librados: init done 2023-08-22T23:44:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:44:50.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.958+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a541d9f50 con 0x7f3a5413e860 2023-08-22T23:44:50.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.978+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3a4401b3b0 con 0x7f3a5413e860 2023-08-22T23:44:50.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.979+0000 7f3a4bfff640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806ba30 0x7f3a3806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:44:50.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.979+0000 7f3a4a7fc640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806ba30 0x7f3a3806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:44:50.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.979+0000 7f3a4a7fc640 10 monclient: get_auth_request con 0x7f3a3806ba30 auth_method 0 2023-08-22T23:44:50.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.980+0000 7f3a4a7fc640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806ba30 0x7f3a3806def0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c035000 tx=0x7f3a3c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:44:50.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:50.980+0000 7f3a4bfff640 10 client.14835.objecter ms_handle_connect 0x7f3a3806ba30 2023-08-22T23:44:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f3a4405b050 con 0x7f3a5413e860 2023-08-22T23:44:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 10 client.14835.objecter ms_dispatch 0x7f3a54000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:44:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 3 client.14835.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:44:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 3 client.14835.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:44:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 20 client.14835.objecter dump_active .. 0 homeless 2023-08-22T23:44:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f3a44057110 con 0x7f3a5413e860 2023-08-22T23:44:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:44:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.242+0000 7f3a4bfff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:44:51.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.468+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a540a9e40 con 0x7f3a3806ba30 2023-08-22T23:44:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.473+0000 7f3a4bfff640 1 -- 172.21.15.29:0/3738137324 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150584 (secure 0 0 0) 0x7f3a3c041070 con 0x7f3a3806ba30 2023-08-22T23:44:51.475 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:44:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 10 librados: watch_flush enter 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 10 librados: watch_flush exit 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 20 client.14835.objecter shutdown clearing up homeless session... 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 10 client.14835.objecter successfully canceled tick 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806ba30 msgr2=0x7f3a3806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806ba30 0x7f3a3806def0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c035000 tx=0x7f3a3c008030 comp rx=0 tx=0).stop 2023-08-22T23:44:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 10 monclient: shutdown 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 msgr2=0x7f3a541d95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.475+0000 7f3a5933d640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 0x7f3a541d95e0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c480 tx=0x7f3a4400c940 comp rx=0 tx=0).stop 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 shutdown_connections 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f3a3806ba30 0x7f3a3806def0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f3a54149a90 0x7f3a541402d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a54144710 0x7f3a5413fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 --2- 172.21.15.29:0/3738137324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5413e860 0x7f3a541d95e0 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:44:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 >> 172.21.15.29:0/3738137324 conn(0x7f3a54065f10 msgr2=0x7f3a540b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:44:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 shutdown_connections 2023-08-22T23:44:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 -- 172.21.15.29:0/3738137324 wait complete. 2023-08-22T23:44:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:44:51.476+0000 7f3a5933d640 1 librados: shutdown 2023-08-22T23:44:51.488 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2023-08-22T23:44:50.575869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":612804,"kb_used_data":100024,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754361916,"statfs":{"total":773094113280,"available":772466601984,"internally_reserved":0,"allocated":102424576,"data_stored":62259580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.814169"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.396197+0000","last_change":"2023-08-22T23:23:40.393453+0000","last_active":"2023-08-22T23:44:40.396197+0000","last_peered":"2023-08-22T23:44:40.396197+0000","last_clean":"2023-08-22T23:44:40.396197+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:40.396197+0000","last_undegraded":"2023-08-22T23:44:40.396197+0000","last_fullsized":"2023-08-22T23:44:40.396197+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:46:36.288105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":868,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.391967+0000","last_change":"2023-08-22T23:23:32.307778+0000","last_active":"2023-08-22T23:44:40.391967+0000","last_peered":"2023-08-22T23:44:40.391967+0000","last_clean":"2023-08-22T23:44:40.391967+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:44:40.391967+0000","last_undegraded":"2023-08-22T23:44:40.391967+0000","last_fullsized":"2023-08-22T23:44:40.391967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:25:17.589038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.857092+0000","last_change":"2023-08-22T23:23:41.258175+0000","last_active":"2023-08-22T23:44:39.857092+0000","last_peered":"2023-08-22T23:44:39.857092+0000","last_clean":"2023-08-22T23:44:39.857092+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:44:39.857092+0000","last_undegraded":"2023-08-22T23:44:39.857092+0000","last_fullsized":"2023-08-22T23:44:39.857092+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.417086+0000","last_change":"2023-08-22T23:23:40.333373+0000","last_active":"2023-08-22T23:44:40.417086+0000","last_peered":"2023-08-22T23:44:40.417086+0000","last_clean":"2023-08-22T23:44:40.417086+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:40.417086+0000","last_undegraded":"2023-08-22T23:44:40.417086+0000","last_fullsized":"2023-08-22T23:44:40.417086+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":876,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.778837+0000","last_change":"2023-08-22T23:23:35.301428+0000","last_active":"2023-08-22T23:44:37.778837+0000","last_peered":"2023-08-22T23:44:37.778837+0000","last_clean":"2023-08-22T23:44:37.778837+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:44:37.778837+0000","last_undegraded":"2023-08-22T23:44:37.778837+0000","last_fullsized":"2023-08-22T23:44:37.778837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:04:58.620425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":871,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.411038+0000","last_change":"2023-08-22T23:23:42.559066+0000","last_active":"2023-08-22T23:44:40.411038+0000","last_peered":"2023-08-22T23:44:40.411038+0000","last_clean":"2023-08-22T23:44:40.411038+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:40.411038+0000","last_undegraded":"2023-08-22T23:44:40.411038+0000","last_fullsized":"2023-08-22T23:44:40.411038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":875,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.749194+0000","last_change":"2023-08-22T23:23:39.799381+0000","last_active":"2023-08-22T23:44:37.749194+0000","last_peered":"2023-08-22T23:44:37.749194+0000","last_clean":"2023-08-22T23:44:37.749194+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:37.749194+0000","last_undegraded":"2023-08-22T23:44:37.749194+0000","last_fullsized":"2023-08-22T23:44:37.749194+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:45:47.412302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.372331+0000","last_change":"2023-08-22T23:23:31.473444+0000","last_active":"2023-08-22T23:44:40.372331+0000","last_peered":"2023-08-22T23:44:40.372331+0000","last_clean":"2023-08-22T23:44:40.372331+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:40.372331+0000","last_undegraded":"2023-08-22T23:44:40.372331+0000","last_fullsized":"2023-08-22T23:44:40.372331+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:41:55.891550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":865,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.397897+0000","last_change":"2023-08-22T23:23:30.881230+0000","last_active":"2023-08-22T23:44:40.397897+0000","last_peered":"2023-08-22T23:44:40.397897+0000","last_clean":"2023-08-22T23:44:40.397897+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:44:40.397897+0000","last_undegraded":"2023-08-22T23:44:40.397897+0000","last_fullsized":"2023-08-22T23:44:40.397897+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:45:22.079165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.423775+0000","last_change":"2023-08-22T23:23:57.358686+0000","last_active":"2023-08-22T23:44:40.423775+0000","last_peered":"2023-08-22T23:44:40.423775+0000","last_clean":"2023-08-22T23:44:40.423775+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:40.423775+0000","last_undegraded":"2023-08-22T23:44:40.423775+0000","last_fullsized":"2023-08-22T23:44:40.423775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":873,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.541242+0000","last_change":"2023-08-22T23:44:48.542439+0000","last_active":"2023-08-22T23:44:48.541242+0000","last_peered":"2023-08-22T23:44:48.541242+0000","last_clean":"2023-08-22T23:44:48.541242+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:48.541242+0000","last_undegraded":"2023-08-22T23:44:48.541242+0000","last_fullsized":"2023-08-22T23:44:48.541242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.539506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:42:32.540349+0000","scrub_duration":0.055977747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.984177+0000","last_change":"2023-08-22T23:23:45.579934+0000","last_active":"2023-08-22T23:44:39.984177+0000","last_peered":"2023-08-22T23:44:39.984177+0000","last_clean":"2023-08-22T23:44:39.984177+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:44:39.984177+0000","last_undegraded":"2023-08-22T23:44:39.984177+0000","last_fullsized":"2023-08-22T23:44:39.984177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":888,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.875089+0000","last_change":"2023-08-22T23:22:30.241386+0000","last_active":"2023-08-22T23:44:39.875089+0000","last_peered":"2023-08-22T23:44:39.875089+0000","last_clean":"2023-08-22T23:44:39.875089+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:44:39.875089+0000","last_undegraded":"2023-08-22T23:44:39.875089+0000","last_fullsized":"2023-08-22T23:44:39.875089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":875,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.813740+0000","last_change":"2023-08-22T23:23:37.170814+0000","last_active":"2023-08-22T23:44:37.813740+0000","last_peered":"2023-08-22T23:44:37.813740+0000","last_clean":"2023-08-22T23:44:37.813740+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:37.813740+0000","last_undegraded":"2023-08-22T23:44:37.813740+0000","last_fullsized":"2023-08-22T23:44:37.813740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:09:15.576464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":880,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.811914+0000","last_change":"2023-08-22T23:22:30.276299+0000","last_active":"2023-08-22T23:44:37.811914+0000","last_peered":"2023-08-22T23:44:37.811914+0000","last_clean":"2023-08-22T23:44:37.811914+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:37.811914+0000","last_undegraded":"2023-08-22T23:44:37.811914+0000","last_fullsized":"2023-08-22T23:44:37.811914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-23T23:36:37.347305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":876,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.835473+0000","last_change":"2023-08-22T23:23:32.260717+0000","last_active":"2023-08-22T23:44:37.835473+0000","last_peered":"2023-08-22T23:44:37.835473+0000","last_clean":"2023-08-22T23:44:37.835473+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:37.835473+0000","last_undegraded":"2023-08-22T23:44:37.835473+0000","last_fullsized":"2023-08-22T23:44:37.835473+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:34:26.225961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":888,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.868749+0000","last_change":"2023-08-22T23:22:30.036499+0000","last_active":"2023-08-22T23:44:39.868749+0000","last_peered":"2023-08-22T23:44:39.868749+0000","last_clean":"2023-08-22T23:44:39.868749+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:44:39.868749+0000","last_undegraded":"2023-08-22T23:44:39.868749+0000","last_fullsized":"2023-08-22T23:44:39.868749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":861,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.176692+0000","last_change":"2023-08-22T23:23:47.632982+0000","last_active":"2023-08-22T23:44:39.176692+0000","last_peered":"2023-08-22T23:44:39.176692+0000","last_clean":"2023-08-22T23:44:39.176692+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:39.176692+0000","last_undegraded":"2023-08-22T23:44:39.176692+0000","last_fullsized":"2023-08-22T23:44:39.176692+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:55:33.947607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":871,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.389017+0000","last_change":"2023-08-22T23:22:30.442903+0000","last_active":"2023-08-22T23:44:40.389017+0000","last_peered":"2023-08-22T23:44:40.389017+0000","last_clean":"2023-08-22T23:44:40.389017+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:40.389017+0000","last_undegraded":"2023-08-22T23:44:40.389017+0000","last_fullsized":"2023-08-22T23:44:40.389017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:53:41.101184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.419229+0000","last_change":"2023-08-22T23:23:44.752571+0000","last_active":"2023-08-22T23:44:40.419229+0000","last_peered":"2023-08-22T23:44:40.419229+0000","last_clean":"2023-08-22T23:44:40.419229+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:40.419229+0000","last_undegraded":"2023-08-22T23:44:40.419229+0000","last_fullsized":"2023-08-22T23:44:40.419229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":867,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.433957+0000","last_change":"2023-08-22T23:23:43.800737+0000","last_active":"2023-08-22T23:44:40.433957+0000","last_peered":"2023-08-22T23:44:40.433957+0000","last_clean":"2023-08-22T23:44:40.433957+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:40.433957+0000","last_undegraded":"2023-08-22T23:44:40.433957+0000","last_fullsized":"2023-08-22T23:44:40.433957+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":888,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.854779+0000","last_change":"2023-08-22T23:22:30.393248+0000","last_active":"2023-08-22T23:44:39.854779+0000","last_peered":"2023-08-22T23:44:39.854779+0000","last_clean":"2023-08-22T23:44:39.854779+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:39.854779+0000","last_undegraded":"2023-08-22T23:44:39.854779+0000","last_fullsized":"2023-08-22T23:44:39.854779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":869,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.369500+0000","last_change":"2023-08-22T23:23:31.308224+0000","last_active":"2023-08-22T23:44:40.369500+0000","last_peered":"2023-08-22T23:44:40.369500+0000","last_clean":"2023-08-22T23:44:40.369500+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:40.369500+0000","last_undegraded":"2023-08-22T23:44:40.369500+0000","last_fullsized":"2023-08-22T23:44:40.369500+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:16:42.335154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":880,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.923274+0000","last_change":"2023-08-22T23:22:31.576502+0000","last_active":"2023-08-22T23:44:38.923274+0000","last_peered":"2023-08-22T23:44:38.923274+0000","last_clean":"2023-08-22T23:44:38.923274+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:38.923274+0000","last_undegraded":"2023-08-22T23:44:38.923274+0000","last_fullsized":"2023-08-22T23:44:38.923274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:26:48.005613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"101'7","reported_seq":874,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.172438+0000","last_change":"2023-08-22T23:23:56.020996+0000","last_active":"2023-08-22T23:44:39.172438+0000","last_peered":"2023-08-22T23:44:39.172438+0000","last_clean":"2023-08-22T23:44:39.172438+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:39.172438+0000","last_undegraded":"2023-08-22T23:44:39.172438+0000","last_fullsized":"2023-08-22T23:44:39.172438+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:06:07.542557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":890,"reported_epoch":918,"state":"active+clean+scrubbing+deep","last_fresh":"2023-08-22T23:44:47.319026+0000","last_change":"2023-08-22T23:44:49.986898+0000","last_active":"2023-08-22T23:44:47.319026+0000","last_peered":"2023-08-22T23:44:47.319026+0000","last_clean":"2023-08-22T23:44:47.319026+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:47.319026+0000","last_undegraded":"2023-08-22T23:44:47.319026+0000","last_fullsized":"2023-08-22T23:44:47.319026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":878,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.791940+0000","last_change":"2023-08-22T23:23:34.106212+0000","last_active":"2023-08-22T23:44:37.791940+0000","last_peered":"2023-08-22T23:44:37.791940+0000","last_clean":"2023-08-22T23:44:37.791940+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:37.791940+0000","last_undegraded":"2023-08-22T23:44:37.791940+0000","last_fullsized":"2023-08-22T23:44:37.791940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:18:39.449835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":888,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.849220+0000","last_change":"2023-08-22T23:22:30.329629+0000","last_active":"2023-08-22T23:44:39.849220+0000","last_peered":"2023-08-22T23:44:39.849220+0000","last_clean":"2023-08-22T23:44:39.849220+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:39.849220+0000","last_undegraded":"2023-08-22T23:44:39.849220+0000","last_fullsized":"2023-08-22T23:44:39.849220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:28.327830+0000","last_clean_scrub_stamp":"2023-08-22T23:22:28.327830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":984,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.403119+0000","last_change":"2023-08-22T23:22:24.344727+0000","last_active":"2023-08-22T23:44:40.403119+0000","last_peered":"2023-08-22T23:44:40.403119+0000","last_clean":"2023-08-22T23:44:40.403119+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:40.403119+0000","last_undegraded":"2023-08-22T23:44:40.403119+0000","last_fullsized":"2023-08-22T23:44:40.403119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:22:21.305061+0000","last_clean_scrub_stamp":"2023-08-22T23:22:21.305061+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":877,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-08-22T23:44:37.733321+0000","last_change":"2023-08-22T23:23:37.130773+0000","last_active":"2023-08-22T23:44:37.733321+0000","last_peered":"2023-08-22T23:44:37.733321+0000","last_clean":"2023-08-22T23:44:37.733321+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:44:37.733321+0000","last_undegraded":"2023-08-22T23:44:37.733321+0000","last_fullsized":"2023-08-22T23:44:37.733321+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:03:26.536335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.843354+0000","last_change":"2023-08-22T23:23:33.761439+0000","last_active":"2023-08-22T23:44:39.843354+0000","last_peered":"2023-08-22T23:44:39.843354+0000","last_clean":"2023-08-22T23:44:39.843354+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:44:39.843354+0000","last_undegraded":"2023-08-22T23:44:39.843354+0000","last_fullsized":"2023-08-22T23:44:39.843354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.851266+0000","last_change":"2023-08-22T23:23:33.024596+0000","last_active":"2023-08-22T23:44:39.851266+0000","last_peered":"2023-08-22T23:44:39.851266+0000","last_clean":"2023-08-22T23:44:39.851266+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:44:39.851266+0000","last_undegraded":"2023-08-22T23:44:39.851266+0000","last_fullsized":"2023-08-22T23:44:39.851266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":875,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:38.926226+0000","last_change":"2023-08-22T23:23:36.066660+0000","last_active":"2023-08-22T23:44:38.926226+0000","last_peered":"2023-08-22T23:44:38.926226+0000","last_clean":"2023-08-22T23:44:38.926226+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:38.926226+0000","last_undegraded":"2023-08-22T23:44:38.926226+0000","last_fullsized":"2023-08-22T23:44:38.926226+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:53:08.491032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:46.514602+0000","last_change":"2023-08-22T23:44:46.515850+0000","last_active":"2023-08-22T23:44:46.514602+0000","last_peered":"2023-08-22T23:44:46.514602+0000","last_clean":"2023-08-22T23:44:46.514602+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:46.514602+0000","last_undegraded":"2023-08-22T23:44:46.514602+0000","last_fullsized":"2023-08-22T23:44:46.514602+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_clean_scrub_stamp":"2023-08-22T23:44:46.513081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:49:45.949593+0000","scrub_duration":0.047647529000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.867039+0000","last_change":"2023-08-22T23:23:32.954050+0000","last_active":"2023-08-22T23:44:39.867039+0000","last_peered":"2023-08-22T23:44:39.867039+0000","last_clean":"2023-08-22T23:44:39.867039+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:44:39.867039+0000","last_undegraded":"2023-08-22T23:44:39.867039+0000","last_fullsized":"2023-08-22T23:44:39.867039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.426289+0000","last_change":"2023-08-22T23:23:55.264729+0000","last_active":"2023-08-22T23:44:40.426289+0000","last_peered":"2023-08-22T23:44:40.426289+0000","last_clean":"2023-08-22T23:44:40.426289+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:40.426289+0000","last_undegraded":"2023-08-22T23:44:40.426289+0000","last_fullsized":"2023-08-22T23:44:40.426289+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":866,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.449757+0000","last_change":"2023-08-22T23:23:39.717617+0000","last_active":"2023-08-22T23:44:40.449757+0000","last_peered":"2023-08-22T23:44:40.449757+0000","last_clean":"2023-08-22T23:44:40.449757+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:40.449757+0000","last_undegraded":"2023-08-22T23:44:40.449757+0000","last_fullsized":"2023-08-22T23:44:40.449757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":872,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:40.407138+0000","last_change":"2023-08-22T23:23:43.444588+0000","last_active":"2023-08-22T23:44:40.407138+0000","last_peered":"2023-08-22T23:44:40.407138+0000","last_clean":"2023-08-22T23:44:40.407138+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:40.407138+0000","last_undegraded":"2023-08-22T23:44:40.407138+0000","last_fullsized":"2023-08-22T23:44:40.407138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.839654+0000","last_change":"2023-08-22T23:23:45.510476+0000","last_active":"2023-08-22T23:44:39.839654+0000","last_peered":"2023-08-22T23:44:39.839654+0000","last_clean":"2023-08-22T23:44:39.839654+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:44:39.839654+0000","last_undegraded":"2023-08-22T23:44:39.839654+0000","last_fullsized":"2023-08-22T23:44:39.839654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":860,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.184556+0000","last_change":"2023-08-22T23:23:33.850083+0000","last_active":"2023-08-22T23:44:39.184556+0000","last_peered":"2023-08-22T23:44:39.184556+0000","last_clean":"2023-08-22T23:44:39.184556+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:39.184556+0000","last_undegraded":"2023-08-22T23:44:39.184556+0000","last_fullsized":"2023-08-22T23:44:39.184556+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:48:04.458489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.983172+0000","last_change":"2023-08-22T23:23:41.098969+0000","last_active":"2023-08-22T23:44:39.983172+0000","last_peered":"2023-08-22T23:44:39.983172+0000","last_clean":"2023-08-22T23:44:39.983172+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:44:39.983172+0000","last_undegraded":"2023-08-22T23:44:39.983172+0000","last_fullsized":"2023-08-22T23:44:39.983172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477006,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66960,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94304880,"statfs":{"total":96636764160,"available":96568197120,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":13.682,"5min":16.966000000000001,"15min":19.555},"min":{"1min":7.7030000000000003,"5min":4.734,"15min":4.734},"max":{"1min":40.639000000000003,"5min":187.08799999999999,"15min":254.63},"last":10.619},{"interface":"front","average":{"1min":17.207999999999998,"5min":23.309999999999999,"15min":22.024999999999999},"min":{"1min":6.1360000000000001,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":47.487000000000002,"5min":330.00099999999998,"15min":330.00099999999998},"last":9.3019999999999996}]},{"osd":2,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":22.524999999999999,"15min":22.077000000000002},"min":{"1min":6.8159999999999998,"5min":6.8099999999999996,"15min":6.3019999999999996},"max":{"1min":52.658000000000001,"5min":176.803,"15min":176.803},"last":16.007999999999999},{"interface":"front","average":{"1min":33.201000000000001,"5min":28.675999999999998,"15min":25.103000000000002},"min":{"1min":8.7319999999999993,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":267.399,"5min":277.048,"15min":277.048},"last":12.204000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.978000000000002,"5min":15.691000000000001,"15min":19.866},"min":{"1min":7.7519999999999998,"5min":6.3860000000000001,"15min":5.766},"max":{"1min":112.148,"5min":112.148,"15min":332.09199999999998},"last":8.9450000000000003},{"interface":"front","average":{"1min":18.84,"5min":24.292000000000002,"15min":25.629000000000001},"min":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":75.138000000000005,"5min":193.38499999999999,"15min":218.94900000000001},"last":15.385}]},{"osd":4,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":21.919,"5min":22.414999999999999,"15min":25.763999999999999},"min":{"1min":9.3780000000000001,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":107.697,"5min":107.697,"15min":333.517},"last":14.73},{"interface":"front","average":{"1min":21.629000000000001,"5min":28.734000000000002,"15min":29.236000000000001},"min":{"1min":9.7230000000000008,"5min":9.1370000000000005,"15min":8.5779999999999994},"max":{"1min":108.776,"5min":273.65800000000002,"15min":342.31999999999999},"last":11.574999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":22.91,"5min":21.954999999999998,"15min":23.404},"min":{"1min":11.204000000000001,"5min":6.0869999999999997,"15min":6.0869999999999997},"max":{"1min":73.780000000000001,"5min":106.52,"15min":326.71300000000002},"last":19.681999999999999},{"interface":"front","average":{"1min":20.088999999999999,"5min":24.102,"15min":27.399999999999999},"min":{"1min":7.4470000000000001,"5min":7.4470000000000001,"15min":7.2830000000000004},"max":{"1min":99.808000000000007,"5min":196.352,"15min":250.43600000000001},"last":18.588999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.93,"5min":23.151,"15min":26.940999999999999},"min":{"1min":10.054,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":50.091999999999999,"5min":110.843,"15min":267.84800000000001},"last":16.632999999999999},{"interface":"front","average":{"1min":24.856000000000002,"5min":30.155999999999999,"15min":29.498999999999999},"min":{"1min":12.632,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":103.387,"5min":198.358,"15min":255.58699999999999},"last":20.965}]},{"osd":7,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":23.925999999999998,"5min":33.804000000000002,"15min":32.649000000000001},"min":{"1min":13.246,"5min":11.016,"15min":10.459},"max":{"1min":76.736999999999995,"5min":274.60899999999998,"15min":274.60899999999998},"last":21.936},{"interface":"front","average":{"1min":22.082000000000001,"5min":22.986999999999998,"15min":24.119},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":7.2030000000000003},"max":{"1min":98.132999999999996,"5min":114.447,"15min":213.56899999999999},"last":13.782}]}]},{"osd":3,"up_from":16,"seq":68719477008,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84604,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287236,"statfs":{"total":96636764160,"available":96550129664,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":17.213000000000001,"5min":15.962999999999999,"15min":21.890000000000001},"min":{"1min":6.5060000000000002,"5min":5.2729999999999997,"15min":5.0359999999999996},"max":{"1min":122.562,"5min":153.98599999999999,"15min":274.45100000000002},"last":10.005000000000001},{"interface":"front","average":{"1min":27.960000000000001,"5min":23.204999999999998,"15min":29.393999999999998},"min":{"1min":9.2579999999999991,"5min":7.4489999999999998,"15min":5.2670000000000003},"max":{"1min":215.34800000000001,"5min":215.34800000000001,"15min":356.75599999999997},"last":16.829999999999998}]},{"osd":1,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":13.138999999999999,"5min":16.411999999999999,"15min":25.859999999999999},"min":{"1min":5.218,"5min":5.141,"15min":5.141},"max":{"1min":56.652999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":19.856999999999999},{"interface":"front","average":{"1min":14.827999999999999,"5min":16.405000000000001,"15min":24.341000000000001},"min":{"1min":6.8529999999999998,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":51.414999999999999,"5min":190.49799999999999,"15min":374.01799999999997},"last":14.435}]},{"osd":2,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":23.094000000000001,"5min":21.440000000000001,"15min":27.952000000000002},"min":{"1min":6.6280000000000001,"5min":6.524,"15min":6.524},"max":{"1min":145.535,"5min":198.649,"15min":353.66000000000003},"last":15.085000000000001},{"interface":"front","average":{"1min":20.449999999999999,"5min":21.263999999999999,"15min":30.486999999999998},"min":{"1min":7.3970000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":108.08,"5min":193.53999999999999,"15min":385.17599999999999},"last":11.816000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.568000000000001,"5min":22.332999999999998,"15min":30.498999999999999},"min":{"1min":9.4250000000000007,"5min":7.2000000000000002,"15min":6.6230000000000002},"max":{"1min":86.454999999999998,"5min":199.42400000000001,"15min":370.84399999999999},"last":11.486000000000001},{"interface":"front","average":{"1min":19.699000000000002,"5min":25.030000000000001,"15min":35.835000000000001},"min":{"1min":8.6219999999999999,"5min":6.726,"15min":6.726},"max":{"1min":64.091999999999999,"5min":231.53,"15min":372.45299999999997},"last":10.808}]},{"osd":5,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":19.824999999999999,"5min":22.189,"15min":32.162999999999997},"min":{"1min":9.7609999999999992,"5min":9.5820000000000007,"15min":8.5419999999999998},"max":{"1min":52.338999999999999,"5min":175.429,"15min":369.33499999999998},"last":15.714},{"interface":"front","average":{"1min":18.379999999999999,"5min":21.294,"15min":29.463000000000001},"min":{"1min":7.9320000000000004,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":54.829000000000001,"5min":212.09,"15min":289.34300000000002},"last":20.788}]},{"osd":6,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":18.791,"5min":22.777999999999999,"15min":33.335999999999999},"min":{"1min":8.8979999999999997,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":57.430999999999997,"5min":213.15000000000001,"15min":372.63799999999998},"last":16.093},{"interface":"front","average":{"1min":24.143999999999998,"5min":27.623999999999999,"15min":31.969000000000001},"min":{"1min":14.032,"5min":8.1010000000000009,"15min":7.7830000000000004},"max":{"1min":63.143000000000001,"5min":215.41300000000001,"15min":346.69900000000001},"last":17.523}]},{"osd":7,"last update":"Tue Aug 22 23:43:58 2023","interfaces":[{"interface":"back","average":{"1min":20.388000000000002,"5min":23.713999999999999,"15min":33.411000000000001},"min":{"1min":10.506,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":58.783999999999999,"5min":206.708,"15min":417.863},"last":21.756},{"interface":"front","average":{"1min":23.991,"5min":28.187999999999999,"15min":31.818999999999999},"min":{"1min":13.135999999999999,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":61.673000000000002,"5min":229.85900000000001,"15min":241.38200000000001},"last":18.216000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477006,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66560,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305280,"statfs":{"total":96636764160,"available":96568606720,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":8.4800000000000004,"5min":16.053999999999998,"15min":17.029},"min":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":15.577999999999999,"5min":204.17500000000001,"15min":208.114},"last":8.5779999999999994},{"interface":"front","average":{"1min":16.081,"5min":21.925999999999998,"15min":20.050000000000001},"min":{"1min":8.7599999999999998,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":44.661000000000001,"5min":205.833,"15min":205.833},"last":9.9220000000000006}]},{"osd":1,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":17.501000000000001,"15min":18.324999999999999},"min":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":114.821,"5min":142.42599999999999,"15min":218.06200000000001},"last":15.73},{"interface":"front","average":{"1min":24.239999999999998,"5min":22.905000000000001,"15min":21.925999999999998},"min":{"1min":10.121,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":115.809,"5min":145.83000000000001,"15min":230.108},"last":18.111000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:43:54 2023","interfaces":[{"interface":"back","average":{"1min":16.317,"5min":21.879000000000001,"15min":21.068999999999999},"min":{"1min":7.0880000000000001,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":102.65300000000001,"5min":292.49000000000001,"15min":292.49000000000001},"last":7.484},{"interface":"front","average":{"1min":20.469999999999999,"5min":28.943999999999999,"15min":25.609000000000002},"min":{"1min":9.5030000000000001,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":107.33799999999999,"5min":304.017,"15min":306.26299999999998},"last":10.77}]},{"osd":4,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":15.987,"5min":19.297999999999998,"15min":21.773},"min":{"1min":8.7170000000000005,"5min":8.3759999999999994,"15min":7.3869999999999996},"max":{"1min":52.591999999999999,"5min":157.559,"15min":187.357},"last":16.888999999999999},{"interface":"front","average":{"1min":14.167999999999999,"5min":18.696999999999999,"15min":20.98},"min":{"1min":7.7050000000000001,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":19.834,"5min":146.64599999999999,"15min":206.59100000000001},"last":14.632}]},{"osd":5,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.920000000000002,"5min":22.468,"15min":24.614000000000001},"min":{"1min":10.135,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":54.234000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":19.305},{"interface":"front","average":{"1min":22.023,"5min":21.733000000000001,"15min":23.268000000000001},"min":{"1min":8.9309999999999992,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":63.061999999999998,"5min":181.24199999999999,"15min":181.24199999999999},"last":13.273999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":27.670999999999999,"5min":25.062999999999999,"15min":24.218},"min":{"1min":11.868,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":127.512,"5min":189.55000000000001,"15min":189.55000000000001},"last":20.515999999999998},{"interface":"front","average":{"1min":25.172999999999998,"5min":21.625,"15min":24.170000000000002},"min":{"1min":9.8800000000000008,"5min":9.0280000000000005,"15min":8.8819999999999997},"max":{"1min":119.714,"5min":129.928,"15min":187.434},"last":13.868}]},{"osd":7,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":26.472999999999999,"15min":29.672000000000001},"min":{"1min":11.529999999999999,"5min":10.513,"15min":9.9939999999999998},"max":{"1min":55.665999999999997,"5min":195.655,"15min":402.06099999999998},"last":21.722000000000001},{"interface":"front","average":{"1min":17.021000000000001,"5min":21.558,"15min":28.571999999999999},"min":{"1min":8.9019999999999992,"5min":8.6690000000000005,"15min":8.6690000000000005},"max":{"1min":39.817,"5min":158.26499999999999,"15min":427.274},"last":12.673}]}]},{"osd":0,"up_from":16,"seq":68719477008,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66152,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305688,"statfs":{"total":96636764160,"available":96569024512,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.164,"15min":17.242999999999999},"min":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":28.050000000000001,"5min":125.212,"15min":315.35300000000001},"last":12.411},{"interface":"front","average":{"1min":11.481999999999999,"5min":19.741,"15min":19.794},"min":{"1min":5.8739999999999997,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":27.388000000000002,"5min":150.68199999999999,"15min":314.375},"last":18.509}]},{"osd":2,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":15.347,"5min":22.693000000000001,"15min":21.102},"min":{"1min":7.5519999999999996,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":42.912999999999997,"5min":133.678,"15min":255.041},"last":13.747},{"interface":"front","average":{"1min":14.930999999999999,"5min":23.047000000000001,"15min":21.428000000000001},"min":{"1min":7.1859999999999999,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":31.559999999999999,"5min":124.077,"15min":266.08600000000001},"last":8.5120000000000005}]},{"osd":3,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":14.122,"5min":21.184999999999999,"15min":20.628},"min":{"1min":7.6429999999999998,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":77.963999999999999,"5min":140.357,"15min":359.37},"last":8.8320000000000007},{"interface":"front","average":{"1min":20.382000000000001,"5min":23.077999999999999,"15min":22.991},"min":{"1min":8.3200000000000003,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":80.066999999999993,"5min":133.27000000000001,"15min":335.35399999999998},"last":10.978}]},{"osd":4,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":17.265999999999998,"5min":21.844999999999999,"15min":22.927},"min":{"1min":9.0359999999999996,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":40.703000000000003,"5min":270.06900000000002,"15min":270.06900000000002},"last":14.385},{"interface":"front","average":{"1min":17.626999999999999,"5min":23.300000000000001,"15min":23.582000000000001},"min":{"1min":7.2880000000000003,"5min":7.1689999999999996,"15min":7.1689999999999996},"max":{"1min":48.825000000000003,"5min":204.54499999999999,"15min":301.78399999999999},"last":11.755000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":18.289999999999999,"5min":24.518000000000001,"15min":25.202999999999999},"min":{"1min":11.141,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":30.463000000000001,"5min":153.78100000000001,"15min":287.80200000000002},"last":16.484000000000002},{"interface":"front","average":{"1min":19.331,"5min":24.251000000000001,"15min":26.376000000000001},"min":{"1min":9.8699999999999992,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":37.241,"5min":130.245,"15min":260.29399999999998},"last":10.371}]},{"osd":6,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":25.445,"5min":28.920999999999999,"15min":25.911000000000001},"min":{"1min":10.363,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":159.46299999999999,"5min":243.15100000000001,"15min":276.935},"last":19.532},{"interface":"front","average":{"1min":21.541,"5min":23.311,"15min":24.834},"min":{"1min":10.583,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":109.886,"5min":241.351,"15min":265.17399999999998},"last":13.141}]},{"osd":7,"last update":"Tue Aug 22 23:44:03 2023","interfaces":[{"interface":"back","average":{"1min":33.301000000000002,"5min":29.783000000000001,"15min":31.905999999999999},"min":{"1min":13.009,"5min":10.702,"15min":7.1760000000000002},"max":{"1min":184.333,"5min":227.852,"15min":770.02200000000005},"last":15.022},{"interface":"front","average":{"1min":31.655000000000001,"5min":27.768000000000001,"15min":31.879999999999999},"min":{"1min":10.598000000000001,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":154.00800000000001,"5min":209.393,"15min":834.71799999999996},"last":18.161999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509712,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66548,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305292,"statfs":{"total":96636764160,"available":96568619008,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":20.427,"5min":17.768000000000001,"15min":18.873999999999999},"min":{"1min":7.3470000000000004,"5min":6.2119999999999997,"15min":4.5570000000000004},"max":{"1min":95.617000000000004,"5min":130.898,"15min":358.06400000000002},"last":7.9249999999999998},{"interface":"front","average":{"1min":24.739999999999998,"5min":24.744,"15min":22.606999999999999},"min":{"1min":10.959,"5min":9.2430000000000003,"15min":4.9699999999999998},"max":{"1min":103.443,"5min":146.55500000000001,"15min":233.333},"last":16.593}]},{"osd":1,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.175999999999998,"5min":15.595000000000001,"15min":17.597000000000001},"min":{"1min":6.4939999999999998,"5min":5.1100000000000003,"15min":5.1100000000000003},"max":{"1min":63.090000000000003,"5min":112.752,"15min":222.346},"last":13.653},{"interface":"front","average":{"1min":20.001999999999999,"5min":19.123000000000001,"15min":20.712},"min":{"1min":8.8339999999999996,"5min":8.0579999999999998,"15min":6.0519999999999996},"max":{"1min":87.179000000000002,"5min":137.86799999999999,"15min":258.90600000000001},"last":15.614000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":21.442,"5min":18.684000000000001,"15min":19.425000000000001},"min":{"1min":6.6619999999999999,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":92.921000000000006,"5min":171.38399999999999,"15min":283.65600000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":22.297999999999998,"5min":21.972000000000001,"15min":23.984999999999999},"min":{"1min":9.0169999999999995,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":100.437,"5min":170.02600000000001,"15min":389.19299999999998},"last":10.347}]},{"osd":3,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":22.475999999999999,"15min":27.465},"min":{"1min":7.6630000000000003,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":208.94800000000001,"5min":214.666,"15min":352.79500000000002},"last":10.987},{"interface":"front","average":{"1min":25.527000000000001,"5min":23.891999999999999,"15min":24.518000000000001},"min":{"1min":10.247999999999999,"5min":7.4370000000000003,"15min":7.415},"max":{"1min":91.753,"5min":178.21600000000001,"15min":314.69},"last":11.391}]},{"osd":4,"last update":"Tue Aug 22 23:43:59 2023","interfaces":[{"interface":"back","average":{"1min":17.849,"5min":20.013000000000002,"15min":23.686},"min":{"1min":7.3520000000000003,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":91.307000000000002,"5min":121.392,"15min":316.47500000000002},"last":14.282},{"interface":"front","average":{"1min":23.972999999999999,"5min":27.207999999999998,"15min":30.826000000000001},"min":{"1min":12.016999999999999,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":92.557000000000002,"5min":156.40100000000001,"15min":326.65100000000001},"last":19.962}]},{"osd":5,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":24.760999999999999,"15min":26.306000000000001},"min":{"1min":10.253,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":86.361999999999995,"5min":183.268,"15min":249.37299999999999},"last":13.138},{"interface":"front","average":{"1min":24.760000000000002,"5min":22.363,"15min":23.788},"min":{"1min":9.9090000000000007,"5min":9.2989999999999995,"15min":7.3849999999999998},"max":{"1min":93.683000000000007,"5min":146.96199999999999,"15min":215.76499999999999},"last":19.108000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.631,"15min":26.940000000000001},"min":{"1min":10.702,"5min":10.388999999999999,"15min":9.3559999999999999},"max":{"1min":46.847999999999999,"5min":135.65700000000001,"15min":457.08199999999999},"last":20.881},{"interface":"front","average":{"1min":19.600999999999999,"5min":26.585999999999999,"15min":26.331},"min":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":54.695,"5min":247.423,"15min":353.75900000000001},"last":17.579000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509711,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88692,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283148,"statfs":{"total":96636764160,"available":96545943552,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":20.602,"5min":22.466999999999999,"15min":22.718},"min":{"1min":4.7880000000000003,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":90.212000000000003,"5min":141.732,"15min":235.06100000000001},"last":7.3099999999999996},{"interface":"front","average":{"1min":25.978000000000002,"5min":24.094000000000001,"15min":27.666},"min":{"1min":7.5999999999999996,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":112.598,"5min":189.517,"15min":395.15300000000002},"last":11.359999999999999}]},{"osd":1,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":36.859999999999999,"5min":25.800999999999998,"15min":26.106999999999999},"min":{"1min":6.9509999999999996,"5min":6.7149999999999999,"15min":5.7439999999999998},"max":{"1min":228.94499999999999,"5min":241.87799999999999,"15min":382.04199999999997},"last":8.5950000000000006},{"interface":"front","average":{"1min":41.487000000000002,"5min":33.701999999999998,"15min":33.277999999999999},"min":{"1min":11.436999999999999,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":237.38499999999999,"5min":252.441,"15min":382.69},"last":12.602}]},{"osd":2,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":23.855,"5min":24.207999999999998,"15min":25.460999999999999},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":214.36500000000001,"5min":219.37899999999999,"15min":537.76199999999994},"last":14.314},{"interface":"front","average":{"1min":20.466999999999999,"5min":26.917999999999999,"15min":32.433999999999997},"min":{"1min":6.5999999999999996,"5min":6.5999999999999996,"15min":6.1210000000000004},"max":{"1min":140.50200000000001,"5min":214.50700000000001,"15min":556.82899999999995},"last":12.000999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":24.536999999999999,"5min":24.959,"15min":28.465},"min":{"1min":8.3970000000000002,"5min":7.0449999999999999,"15min":6.6159999999999997},"max":{"1min":160.98400000000001,"5min":161.05199999999999,"15min":354.50400000000002},"last":15.304},{"interface":"front","average":{"1min":31.042999999999999,"5min":30.719999999999999,"15min":30.981999999999999},"min":{"1min":10.749000000000001,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":226.25700000000001,"5min":226.25700000000001,"15min":339.51499999999999},"last":17.853999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":34.338000000000001,"5min":30.631,"15min":32.762999999999998},"min":{"1min":9.1660000000000004,"5min":7.4160000000000004,"15min":7.2279999999999998},"max":{"1min":139.31899999999999,"5min":238.57300000000001,"15min":380.863},"last":9.5660000000000007},{"interface":"front","average":{"1min":26.352,"5min":30.120999999999999,"15min":36.542000000000002},"min":{"1min":7.5999999999999996,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":99.632999999999996,"5min":215.928,"15min":364.09500000000003},"last":16.245999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":32.222000000000001,"5min":30.013999999999999,"15min":31.196000000000002},"min":{"1min":12.083,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":190.446,"5min":217,"15min":249.29599999999999},"last":18.506},{"interface":"front","average":{"1min":24.103000000000002,"5min":24.334,"15min":31.428000000000001},"min":{"1min":8.952,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":132.864,"5min":220.87100000000001,"15min":468.887},"last":10.704000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.209,"5min":24.561,"15min":30.809000000000001},"min":{"1min":10.443,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":26.222999999999999,"5min":187.30099999999999,"15min":361.45499999999998},"last":19.260999999999999},{"interface":"front","average":{"1min":19.317,"5min":28.436,"15min":31.718},"min":{"1min":14.016999999999999,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":33.298999999999999,"5min":197.05600000000001,"15min":368.08699999999999},"last":20.524000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542417,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88304,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283536,"statfs":{"total":96636764160,"available":96546340864,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":11.916,"5min":19.649000000000001,"15min":20.076000000000001},"min":{"1min":5.6710000000000003,"5min":5.6710000000000003,"15min":4.3040000000000003},"max":{"1min":38.225999999999999,"5min":307.584,"15min":307.584},"last":11.592000000000001},{"interface":"front","average":{"1min":16.166,"5min":21.260999999999999,"15min":20.367000000000001},"min":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":48.941000000000003,"5min":317.81,"15min":317.81},"last":16.161000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":13.314,"5min":17.096,"15min":25.003},"min":{"1min":6.2629999999999999,"5min":5.077,"15min":5.077},"max":{"1min":22.173999999999999,"5min":159.58799999999999,"15min":319.34500000000003},"last":22.248000000000001},{"interface":"front","average":{"1min":11.484999999999999,"5min":14.808,"15min":20.055},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":26.338000000000001,"5min":160.74600000000001,"15min":195.78999999999999},"last":11.951000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.140000000000001,"5min":17.152999999999999,"15min":23.173999999999999},"min":{"1min":8.6620000000000008,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":47.432000000000002,"5min":148.97800000000001,"15min":772.00400000000002},"last":16.895},{"interface":"front","average":{"1min":17.105,"5min":20.027999999999999,"15min":24.41},"min":{"1min":10.882,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":43.933999999999997,"5min":151.46700000000001,"15min":789.553},"last":8.9600000000000009}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.602,"5min":19.146000000000001,"15min":22.295999999999999},"min":{"1min":7.7370000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":70.665999999999997,"5min":154.44200000000001,"15min":243.33699999999999},"last":12.693},{"interface":"front","average":{"1min":18.843,"5min":24.088999999999999,"15min":26.478000000000002},"min":{"1min":6.7880000000000003,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":68.334000000000003,"5min":337.09699999999998,"15min":337.09699999999998},"last":19.356000000000002}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":20.622,"5min":21.59,"15min":29.172999999999998},"min":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.0570000000000004},"max":{"1min":109.88800000000001,"5min":111.015,"15min":333.04000000000002},"last":10.300000000000001},{"interface":"front","average":{"1min":18.895,"5min":20.015999999999998,"15min":24.294},"min":{"1min":8.7140000000000004,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":97.224999999999994,"5min":117.06100000000001,"15min":240.084},"last":15.308999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:05 2023","interfaces":[{"interface":"back","average":{"1min":20.968,"5min":24.206,"15min":30.463999999999999},"min":{"1min":9.9779999999999998,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":60.380000000000003,"5min":247.898,"15min":342.53100000000001},"last":14.618},{"interface":"front","average":{"1min":19.407,"5min":24.077999999999999,"15min":27.297999999999998},"min":{"1min":8.8770000000000007,"5min":8.8770000000000007,"15min":7.4749999999999996},"max":{"1min":49.686,"5min":176.309,"15min":271.47500000000002},"last":17.722999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.318000000000001,"5min":21.977,"15min":25.302},"min":{"1min":8.5459999999999994,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":32.176000000000002,"5min":144.21600000000001,"15min":228.43100000000001},"last":21.308},{"interface":"front","average":{"1min":17.408000000000001,"5min":23.198,"15min":28.206},"min":{"1min":10.394,"5min":10.090999999999999,"15min":9.2650000000000006},"max":{"1min":24.224,"5min":141.92599999999999,"15min":262.45800000000003},"last":20.076000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542414,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84984,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94286856,"statfs":{"total":96636764160,"available":96549740544,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":12.601000000000001,"5min":16.588000000000001,"15min":20.968},"min":{"1min":6.7779999999999996,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":33.573999999999998,"5min":114.51300000000001,"15min":220.017},"last":8.3789999999999996},{"interface":"front","average":{"1min":18.562000000000001,"5min":22.257999999999999,"15min":27.212},"min":{"1min":10.353999999999999,"5min":6.7919999999999998,"15min":5.9329999999999998},"max":{"1min":32.883000000000003,"5min":115.74299999999999,"15min":264.11900000000003},"last":14.071}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":10.41,"5min":14.409000000000001,"15min":20.576000000000001},"min":{"1min":6.5119999999999996,"5min":6.1289999999999996,"15min":5.9379999999999997},"max":{"1min":16.731000000000002,"5min":102.66200000000001,"15min":343.899},"last":11.063000000000001},{"interface":"front","average":{"1min":15.853,"5min":21.071999999999999,"15min":25.216000000000001},"min":{"1min":8.7690000000000001,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":27.713000000000001,"5min":111.873,"15min":270.40499999999997},"last":10.393000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":14.057,"5min":17.219000000000001,"15min":21.003},"min":{"1min":6.4080000000000004,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":28.347000000000001,"5min":116.45,"15min":248.74799999999999},"last":9.6929999999999996},{"interface":"front","average":{"1min":14.875,"5min":17.582000000000001,"15min":24.184999999999999},"min":{"1min":7.0060000000000002,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":26.632000000000001,"5min":114.843,"15min":256.94400000000002},"last":11.891999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":24.103999999999999,"15min":30.831},"min":{"1min":7.6070000000000002,"5min":6.5739999999999998,"15min":6.5739999999999998},"max":{"1min":50.060000000000002,"5min":231.99799999999999,"15min":257.23500000000001},"last":17.760000000000002},{"interface":"front","average":{"1min":25.657,"5min":23.934999999999999,"15min":24.399000000000001},"min":{"1min":9.8130000000000006,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":62.527999999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":16.274999999999999}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.437999999999999,"5min":25.553000000000001,"15min":28.288},"min":{"1min":9.5950000000000006,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":33.143000000000001,"5min":208.03200000000001,"15min":374.83499999999998},"last":19.227},{"interface":"front","average":{"1min":15.750999999999999,"5min":25.452999999999999,"15min":26.808},"min":{"1min":9.1219999999999999,"5min":7.5389999999999997,"15min":6.3019999999999996},"max":{"1min":22.956,"5min":273.58699999999999,"15min":341.255},"last":12.606}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":21.895,"5min":26.039000000000001,"15min":31.167999999999999},"min":{"1min":9.6560000000000006,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":49.250999999999998,"5min":228.626,"15min":241.26499999999999},"last":16.943000000000001},{"interface":"front","average":{"1min":16.486000000000001,"5min":23.117000000000001,"15min":26.573},"min":{"1min":8.891,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":32.026000000000003,"5min":247.06800000000001,"15min":247.06800000000001},"last":19.946999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":21.434000000000001,"15min":27.062999999999999},"min":{"1min":10.872,"5min":8.2560000000000002,"15min":8.2560000000000002},"max":{"1min":31.568000000000001,"5min":121.524,"15min":317.44600000000003},"last":20.643000000000001},{"interface":"front","average":{"1min":17.907,"5min":25.620000000000001,"15min":28.559999999999999},"min":{"1min":11.589,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":30.829000000000001,"5min":201.81700000000001,"15min":361.93000000000001},"last":13.259}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:44:51.495 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.495 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.496 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.496 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.496 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.496 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.497 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.497 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.497 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.497 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.497 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.498 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.498 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.498 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.498 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.498 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.499 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.499 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.499 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.499 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.500 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.500 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.500 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.500 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.501 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.501 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.501 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-08-22T23:22:28.327830+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=28, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.501 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-08-22T23:22:21.305061+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=22, tm_sec=21, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.501 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.502 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.502 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.502 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.502 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.502 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.503 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.503 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.503 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.503 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.504 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:44:51.504 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-08-22T23:44:53.630 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:38.913 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.635 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:38.919 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.649 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:38.933 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.679 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:38.977 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.680 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:38.978 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.691 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:38.989 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.705 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:38.989 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.709 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:38.993 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:44:53.720 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:39.004 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.543 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:48.827 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.547 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:48.831 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.563 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:48.846 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.597 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:48.880 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.597 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:48.894 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.598 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:48.882 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.599 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:48.897 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.613 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:48.911 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:03.614 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:48.898 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:11.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:45:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.616+0000 7f320f342640 10 monclient: get_monmap_and_config 2023-08-22T23:45:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.616+0000 7f320f342640 10 monclient: build_initial_monmap 2023-08-22T23:45:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.616+0000 7f320f342640 10 monclient: monmap: 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:45:11.617518+0000 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:45:11.617518+0000 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:45:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:45:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:45:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:45:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:45:11.619 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.616+0000 7f320f342640 1 Processor -- start 2023-08-22T23:45:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 1 -- start start 2023-08-22T23:45:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient: init 2023-08-22T23:45:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-08-22T23:45:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f32080a8210 0x7f32080a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): picked mon.noname-c con 0x7f32080a8210 addr v2:172.21.15.52:3300/0 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32080b01e0 0x7f3208144690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): picked mon.noname-a con 0x7f32080b01e0 addr v2:172.21.15.29:3300/0 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32080b0600 addr v1:172.21.15.29:6789/0 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32080a7770 con 0x7f32080b0600 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32081452e0 con 0x7f32080b01e0 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32080a8b30 con 0x7f32080a8210 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.617+0000 7f320f342640 10 monclient(hunting): authenticate will time out at 2875.868168s 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32080b01e0 0x7f3208144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320d0de640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f32080a8210 0x7f32080a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32080b01e0 0x7f3208144690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46684/0 (socket says 172.21.15.29:46684) 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 1 -- 172.21.15.29:0/1033509600 learned_addr learned my addr 172.21.15.29:0/1033509600 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 10 monclient(hunting): get_auth_request con 0x7f32080b01e0 auth_method 0 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320c8dd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320d0de640 10 monclient(hunting): get_auth_request con 0x7f32080a8210 auth_method 0 2023-08-22T23:45:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320d0de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320d0de640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.618+0000 7f320d0de640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.619+0000 7f320c8dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.619+0000 7f320c8dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.619+0000 7f320c8dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.619+0000 7f320d0de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.619+0000 7f320d0de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.619+0000 7f320d0de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:11.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 10 monclient(hunting): handle_auth_done global_id 14838 payload 274 2023-08-22T23:45:11.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 1 -- 172.21.15.29:0/1033509600 >> v2:172.21.15.52:3300/0 conn(0x7f32080a8210 msgr2=0x7f32080a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 1 --2- 172.21.15.29:0/1033509600 >> v2:172.21.15.52:3300/0 conn(0x7f32080a8210 0x7f32080a85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 1 -- 172.21.15.29:0/1033509600 >> v1:172.21.15.29:6789/0 conn(0x7f32080b0600 legacy=0x7f3208144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 10 monclient: _finish_hunting 0 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 1 monclient: found mon.noname-a 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.621+0000 7f320c8dd640 1 -- 172.21.15.29:0/1033509600 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32080a8d30 con 0x7f32080b01e0 2023-08-22T23:45:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.623+0000 7f320c8dd640 1 --2- 172.21.15.29:0/1033509600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080b01e0 0x7f3208144690 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f32080b3370 tx=0x7f31f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc05e5195e2bc04 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:11.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 1 -- 172.21.15.29:0/1033509600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8030860 con 0x7f32080b01e0 2023-08-22T23:45:11.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: dump: 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.631 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: _finish_auth 0 2023-08-22T23:45:11.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: _check_auth_tickets 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 1 -- 172.21.15.29:0/1033509600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f80309c0 con 0x7f32080b01e0 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f320f342640 5 monclient: authenticate success, global_id 14838 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f320f342640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 1 -- 172.21.15.29:0/1033509600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8030d50 con 0x7f32080b01e0 2023-08-22T23:45:11.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f320f342640 10 monclient: get_monmap_and_config success 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f320f342640 4 set_mon_vals no callback set 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.629+0000 7f31fffff640 10 monclient: dump: 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:11.635 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 10 monclient: shutdown 2023-08-22T23:45:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 -- 172.21.15.29:0/1033509600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080b01e0 msgr2=0x7f3208144690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 --2- 172.21.15.29:0/1033509600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080b01e0 0x7f3208144690 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f32080b3370 tx=0x7f31f802b9a0 comp rx=0 tx=0).stop 2023-08-22T23:45:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 -- 172.21.15.29:0/1033509600 shutdown_connections 2023-08-22T23:45:11.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 --2- 172.21.15.29:0/1033509600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080b01e0 0x7f3208144690 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 --2- 172.21.15.29:0/1033509600 >> v2:172.21.15.52:3300/0 conn(0x7f32080a8210 0x7f32080a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 -- 172.21.15.29:0/1033509600 >> 172.21.15.29:0/1033509600 conn(0x7f32080b6a10 msgr2=0x7f32080b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 -- 172.21.15.29:0/1033509600 shutdown_connections 2023-08-22T23:45:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.630+0000 7f320f342640 1 -- 172.21.15.29:0/1033509600 wait complete. 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 10 monclient: build_initial_monmap 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 10 monclient: monmap: 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:45:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 1 librados: starting msgr at 2023-08-22T23:45:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 1 librados: starting objecter 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 1 Processor -- start 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 1 -- start start 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 1 librados: setting wanted keys 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 1 librados: calling monclient init 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.631+0000 7f320f342640 10 monclient: init 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient: _add_conns ranks=[0,1,2] 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 0x7f3208136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): picked mon.noname-a con 0x7f32080a8210 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32080b01e0 0x7f3208135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): picked mon.noname-b con 0x7f32080b01e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f32080b0600 0x7f32081d5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): picked mon.noname-c con 0x7f32080b0600 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3208138290 con 0x7f32080a8210 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3208138400 con 0x7f32080b01e0 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3208136070 con 0x7f32080b0600 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320f342640 10 monclient(hunting): authenticate will time out at 2875.882874s 2023-08-22T23:45:11.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320d0de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 0x7f3208136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.632+0000 7f320d8df640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f32080b0600 0x7f32081d5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 0x7f3208136a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46694/0 (socket says 172.21.15.29:46694) 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320c8dd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32080b01e0 0x7f3208135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 1 -- 172.21.15.29:0/2121716784 learned_addr learned my addr 172.21.15.29:0/2121716784 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): get_auth_request con 0x7f32080a8210 auth_method 0 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): get_auth_request con 0x7f32080b0600 auth_method 0 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320c8dd640 10 monclient(hunting): get_auth_request con 0x7f32080b01e0 auth_method 0 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320c8dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320c8dd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:11.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320c8dd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d0de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.633+0000 7f320d8df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.634+0000 7f320c8dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.634+0000 7f320c8dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.634+0000 7f320c8dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 10 monclient(hunting): handle_auth_done global_id 14841 payload 824 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 1 -- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f32080b0600 msgr2=0x7f32081d5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f32080b0600 0x7f32081d5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 1 -- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32080b01e0 msgr2=0x7f3208135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32080b01e0 0x7f3208135b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 10 monclient: _finish_hunting 0 2023-08-22T23:45:11.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 1 monclient: found mon.noname-a 2023-08-22T23:45:11.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:11.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:45:11.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.636+0000 7f320d0de640 1 -- 172.21.15.29:0/2121716784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80094d0 con 0x7f32080a8210 2023-08-22T23:45:11.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.639+0000 7f320d0de640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 0x7f3208136a70 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f320400e9b0 tx=0x7f320400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:11.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.639+0000 7f31fdffb640 10 client.?.objecter ms_handle_connect 0x7f32080a8210 2023-08-22T23:45:11.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.639+0000 7f31fdffb640 10 client.?.objecter resend_mon_ops 2023-08-22T23:45:11.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204009670 con 0x7f32080a8210 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: dump: 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: _finish_auth 0 2023-08-22T23:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: _check_auth_tickets 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320400c9a0 con 0x7f32080a8210 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320400cd50 con 0x7f32080a8210 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 5 monclient: authenticate success, global_id 14841 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 10 monclient: _renew_subs 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32081362f0 con 0x7f32080a8210 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31ff7fe640 4 set_mon_vals no callback set 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f31fdffb640 10 monclient: dump: 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:11.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 10 client.14841.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 10 monclient: _renew_subs 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32081d6570 con 0x7f32080a8210 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.643+0000 7f320f342640 1 librados: init done 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.644+0000 7f320f342640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.644+0000 7f320f342640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.644+0000 7f320f342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:11.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.644+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3208066cb0 con 0x7f32080a8210 2023-08-22T23:45:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.901+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3204013470 con 0x7f32080a8210 2023-08-22T23:45:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.902+0000 7f31fdffb640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f31ec06ba30 0x7f31ec06def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.902+0000 7f320c8dd640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f31ec06ba30 0x7f31ec06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.902+0000 7f320c8dd640 10 monclient: get_auth_request con 0x7f31ec06ba30 auth_method 0 2023-08-22T23:45:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.903+0000 7f320c8dd640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f31ec06ba30 0x7f31ec06def0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f31f8036000 tx=0x7f31f8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.903+0000 7f31fdffb640 10 client.14841.objecter ms_handle_connect 0x7f31ec06ba30 2023-08-22T23:45:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f320405c0a0 con 0x7f32080a8210 2023-08-22T23:45:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 10 client.14841.objecter ms_dispatch 0x7f3208000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:45:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 3 client.14841.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:45:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 3 client.14841.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:45:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 20 client.14841.objecter dump_active .. 0 homeless 2023-08-22T23:45:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7f32040546d0 con 0x7f32080a8210 2023-08-22T23:45:11.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:11.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:11.914+0000 7f31fdffb640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:45:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.096+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32080b2c00 con 0x7f31ec06ba30 2023-08-22T23:45:12.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.101+0000 7f31fdffb640 1 -- 172.21.15.29:0/2121716784 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151659 (secure 0 0 0) 0x7f31f8030bb0 con 0x7f31ec06ba30 2023-08-22T23:45:12.104 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:45:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:45:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 10 librados: watch_flush enter 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 10 librados: watch_flush exit 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 20 client.14841.objecter shutdown clearing up homeless session... 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 10 client.14841.objecter successfully canceled tick 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f31ec06ba30 msgr2=0x7f31ec06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f31ec06ba30 0x7f31ec06def0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f31f8036000 tx=0x7f31f8008030 comp rx=0 tx=0).stop 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 10 monclient: shutdown 2023-08-22T23:45:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 msgr2=0x7f3208136a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 0x7f3208136a70 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f320400e9b0 tx=0x7f320400ee70 comp rx=0 tx=0).stop 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 shutdown_connections 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7f31ec06ba30 0x7f31ec06def0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7f32080b0600 0x7f32081d5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32080b01e0 0x7f3208135b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 --2- 172.21.15.29:0/2121716784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32080a8210 0x7f3208136a70 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 >> 172.21.15.29:0/2121716784 conn(0x7f32080b6a10 msgr2=0x7f32080b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 shutdown_connections 2023-08-22T23:45:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 -- 172.21.15.29:0/2121716784 wait complete. 2023-08-22T23:45:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:12.104+0000 7f320f342640 1 librados: shutdown 2023-08-22T23:45:12.116 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1608,"stamp":"2023-08-22T23:45:10.580670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606456,"kb_used_data":100024,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754368264,"statfs":{"total":773094113280,"available":772473102336,"internally_reserved":0,"allocated":102424576,"data_stored":62259580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:56.865960+0000","last_change":"2023-08-22T23:44:56.867424+0000","last_active":"2023-08-22T23:44:56.865960+0000","last_peered":"2023-08-22T23:44:56.865960+0000","last_clean":"2023-08-22T23:44:56.865960+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:56.865960+0000","last_undegraded":"2023-08-22T23:44:56.865960+0000","last_fullsized":"2023-08-22T23:44:56.865960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:56.864518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:56.864518+0000","last_clean_scrub_stamp":"2023-08-22T23:44:56.864518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:47:28.870179+0000","scrub_duration":0.075777624000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:06.861939+0000","last_change":"2023-08-22T23:45:06.865880+0000","last_active":"2023-08-22T23:45:06.861939+0000","last_peered":"2023-08-22T23:45:06.861939+0000","last_clean":"2023-08-22T23:45:06.861939+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:45:06.861939+0000","last_undegraded":"2023-08-22T23:45:06.861939+0000","last_fullsized":"2023-08-22T23:45:06.861939+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:06.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:06.860450+0000","last_clean_scrub_stamp":"2023-08-22T23:45:06.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:05:25.109783+0000","scrub_duration":0.052525585999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.857092+0000","last_change":"2023-08-22T23:23:41.258175+0000","last_active":"2023-08-22T23:44:39.857092+0000","last_peered":"2023-08-22T23:44:39.857092+0000","last_clean":"2023-08-22T23:44:39.857092+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:44:39.857092+0000","last_undegraded":"2023-08-22T23:44:39.857092+0000","last_fullsized":"2023-08-22T23:44:39.857092+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.370589+0000","last_change":"2023-08-22T23:44:58.375301+0000","last_active":"2023-08-22T23:44:58.370589+0000","last_peered":"2023-08-22T23:44:58.370589+0000","last_clean":"2023-08-22T23:44:58.370589+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:58.370589+0000","last_undegraded":"2023-08-22T23:44:58.370589+0000","last_fullsized":"2023-08-22T23:44:58.370589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.369087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.369087+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.369087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:24:12.816396+0000","scrub_duration":0.031867338000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:01.069233+0000","last_change":"2023-08-22T23:45:01.070353+0000","last_active":"2023-08-22T23:45:01.069233+0000","last_peered":"2023-08-22T23:45:01.069233+0000","last_clean":"2023-08-22T23:45:01.069233+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:45:01.069233+0000","last_undegraded":"2023-08-22T23:45:01.069233+0000","last_fullsized":"2023-08-22T23:45:01.069233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:01.067805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:01.067805+0000","last_clean_scrub_stamp":"2023-08-22T23:45:01.067805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:57:44.262712+0000","scrub_duration":0.080561184999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:57.348370+0000","last_change":"2023-08-22T23:44:57.349504+0000","last_active":"2023-08-22T23:44:57.348370+0000","last_peered":"2023-08-22T23:44:57.348370+0000","last_clean":"2023-08-22T23:44:57.348370+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:57.348370+0000","last_undegraded":"2023-08-22T23:44:57.348370+0000","last_fullsized":"2023-08-22T23:44:57.348370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:57.346862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:57.346862+0000","last_clean_scrub_stamp":"2023-08-22T23:44:57.346862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:08:50.682485+0000","scrub_duration":0.036164746999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.036831+0000","last_change":"2023-08-22T23:44:58.037996+0000","last_active":"2023-08-22T23:44:58.036831+0000","last_peered":"2023-08-22T23:44:58.036831+0000","last_clean":"2023-08-22T23:44:58.036831+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:58.036831+0000","last_undegraded":"2023-08-22T23:44:58.036831+0000","last_fullsized":"2023-08-22T23:44:58.036831+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.035322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.035322+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.035322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:02:25.891429+0000","scrub_duration":0.057636233000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.739887+0000","last_change":"2023-08-22T23:44:52.744778+0000","last_active":"2023-08-22T23:44:52.739887+0000","last_peered":"2023-08-22T23:44:52.739887+0000","last_clean":"2023-08-22T23:44:52.739887+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:52.739887+0000","last_undegraded":"2023-08-22T23:44:52.739887+0000","last_fullsized":"2023-08-22T23:44:52.739887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.738185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.738185+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.738185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:02:04.429354+0000","scrub_duration":0.031314258999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:01.867257+0000","last_change":"2023-08-22T23:45:01.868432+0000","last_active":"2023-08-22T23:45:01.867257+0000","last_peered":"2023-08-22T23:45:01.867257+0000","last_clean":"2023-08-22T23:45:01.867257+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:45:01.867257+0000","last_undegraded":"2023-08-22T23:45:01.867257+0000","last_fullsized":"2023-08-22T23:45:01.867257+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:01.865727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:01.865727+0000","last_clean_scrub_stamp":"2023-08-22T23:45:01.865727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:56:14.967946+0000","scrub_duration":0.082296422999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":890,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.671859+0000","last_change":"2023-08-22T23:44:53.674168+0000","last_active":"2023-08-22T23:44:53.671859+0000","last_peered":"2023-08-22T23:44:53.671859+0000","last_clean":"2023-08-22T23:44:53.671859+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:53.671859+0000","last_undegraded":"2023-08-22T23:44:53.671859+0000","last_fullsized":"2023-08-22T23:44:53.671859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.668393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.668393+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.668393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:07:44.060301+0000","scrub_duration":0.039868937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.542439+0000","last_change":"2023-08-22T23:44:48.542439+0000","last_active":"2023-08-22T23:44:48.542439+0000","last_peered":"2023-08-22T23:44:48.542439+0000","last_clean":"2023-08-22T23:44:48.542439+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:48.542439+0000","last_undegraded":"2023-08-22T23:44:48.542439+0000","last_fullsized":"2023-08-22T23:44:48.542439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.539506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:42:32.540349+0000","scrub_duration":0.055977747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.984177+0000","last_change":"2023-08-22T23:23:45.579934+0000","last_active":"2023-08-22T23:44:39.984177+0000","last_peered":"2023-08-22T23:44:39.984177+0000","last_clean":"2023-08-22T23:44:39.984177+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:44:39.984177+0000","last_undegraded":"2023-08-22T23:44:39.984177+0000","last_fullsized":"2023-08-22T23:44:39.984177+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:09.338967+0000","last_change":"2023-08-22T23:45:09.340155+0000","last_active":"2023-08-22T23:45:09.338967+0000","last_peered":"2023-08-22T23:45:09.338967+0000","last_clean":"2023-08-22T23:45:09.338967+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:45:09.338967+0000","last_undegraded":"2023-08-22T23:45:09.338967+0000","last_fullsized":"2023-08-22T23:45:09.338967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:09.337450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:09.337450+0000","last_clean_scrub_stamp":"2023-08-22T23:45:09.337450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:26:04.335737+0000","scrub_duration":0.031976879,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.033514+0000","last_change":"2023-08-22T23:44:52.037886+0000","last_active":"2023-08-22T23:44:52.033514+0000","last_peered":"2023-08-22T23:44:52.033514+0000","last_clean":"2023-08-22T23:44:52.033514+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:52.033514+0000","last_undegraded":"2023-08-22T23:44:52.033514+0000","last_fullsized":"2023-08-22T23:44:52.033514+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.032042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.032042+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.032042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:04:20.437680+0000","scrub_duration":0.055160028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":889,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.903442+0000","last_change":"2023-08-22T23:44:48.907344+0000","last_active":"2023-08-22T23:44:48.903442+0000","last_peered":"2023-08-22T23:44:48.903442+0000","last_clean":"2023-08-22T23:44:48.903442+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:48.903442+0000","last_undegraded":"2023-08-22T23:44:48.903442+0000","last_fullsized":"2023-08-22T23:44:48.903442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.901951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.901951+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.901951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:56:14.326385+0000","scrub_duration":0.037514621999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.034124+0000","last_change":"2023-08-22T23:44:53.035238+0000","last_active":"2023-08-22T23:44:53.034124+0000","last_peered":"2023-08-22T23:44:53.034124+0000","last_clean":"2023-08-22T23:44:53.034124+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:53.034124+0000","last_undegraded":"2023-08-22T23:44:53.034124+0000","last_fullsized":"2023-08-22T23:44:53.034124+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.032673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.032673+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.032673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:32:17.339073+0000","scrub_duration":0.075927522999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:04.306062+0000","last_change":"2023-08-22T23:45:04.307187+0000","last_active":"2023-08-22T23:45:04.306062+0000","last_peered":"2023-08-22T23:45:04.306062+0000","last_clean":"2023-08-22T23:45:04.306062+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:45:04.306062+0000","last_undegraded":"2023-08-22T23:45:04.306062+0000","last_fullsized":"2023-08-22T23:45:04.306062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:04.304664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:04.304664+0000","last_clean_scrub_stamp":"2023-08-22T23:45:04.304664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:28:11.096423+0000","scrub_duration":0.036375984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":870,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.913093+0000","last_change":"2023-08-22T23:44:58.917636+0000","last_active":"2023-08-22T23:44:58.913093+0000","last_peered":"2023-08-22T23:44:58.913093+0000","last_clean":"2023-08-22T23:44:58.913093+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:58.913093+0000","last_undegraded":"2023-08-22T23:44:58.913093+0000","last_fullsized":"2023-08-22T23:44:58.913093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.911627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.911627+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.911627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:12:11.522411+0000","scrub_duration":0.032531514999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:50.737739+0000","last_change":"2023-08-22T23:44:50.739077+0000","last_active":"2023-08-22T23:44:50.737739+0000","last_peered":"2023-08-22T23:44:50.737739+0000","last_clean":"2023-08-22T23:44:50.737739+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:50.737739+0000","last_undegraded":"2023-08-22T23:44:50.737739+0000","last_fullsized":"2023-08-22T23:44:50.737739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:50.736264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:50.736264+0000","last_clean_scrub_stamp":"2023-08-22T23:44:50.736264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:58:36.038766+0000","scrub_duration":0.044869521000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:47.828710+0000","last_change":"2023-08-22T23:44:47.829841+0000","last_active":"2023-08-22T23:44:47.828710+0000","last_peered":"2023-08-22T23:44:47.828710+0000","last_clean":"2023-08-22T23:44:47.828710+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:47.828710+0000","last_undegraded":"2023-08-22T23:44:47.828710+0000","last_fullsized":"2023-08-22T23:44:47.828710+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:47.827283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:47.827283+0000","last_clean_scrub_stamp":"2023-08-22T23:44:47.827283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:01:52.954243+0000","scrub_duration":0.069352237999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.638075+0000","last_change":"2023-08-22T23:44:51.639230+0000","last_active":"2023-08-22T23:44:51.638075+0000","last_peered":"2023-08-22T23:44:51.638075+0000","last_clean":"2023-08-22T23:44:51.638075+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:51.638075+0000","last_undegraded":"2023-08-22T23:44:51.638075+0000","last_fullsized":"2023-08-22T23:44:51.638075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:51.636674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:51.636674+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.636674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:06:31.966373+0000","scrub_duration":0.052995683000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.068250+0000","last_change":"2023-08-22T23:44:51.072250+0000","last_active":"2023-08-22T23:44:51.068250+0000","last_peered":"2023-08-22T23:44:51.068250+0000","last_clean":"2023-08-22T23:44:51.068250+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:51.068250+0000","last_undegraded":"2023-08-22T23:44:51.068250+0000","last_fullsized":"2023-08-22T23:44:51.068250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:51.066863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:51.066863+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.066863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:51:38.875911+0000","scrub_duration":0.038690479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":878,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.794477+0000","last_change":"2023-08-22T23:44:54.795936+0000","last_active":"2023-08-22T23:44:54.794477+0000","last_peered":"2023-08-22T23:44:54.794477+0000","last_clean":"2023-08-22T23:44:54.794477+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:54.794477+0000","last_undegraded":"2023-08-22T23:44:54.794477+0000","last_fullsized":"2023-08-22T23:44:54.794477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.792995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.792995+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.792995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:03:09.502831+0000","scrub_duration":0.033049222000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":889,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.480298+0000","last_change":"2023-08-22T23:44:52.481421+0000","last_active":"2023-08-22T23:44:52.480298+0000","last_peered":"2023-08-22T23:44:52.480298+0000","last_clean":"2023-08-22T23:44:52.480298+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:52.480298+0000","last_undegraded":"2023-08-22T23:44:52.480298+0000","last_fullsized":"2023-08-22T23:44:52.480298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.478869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.478869+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.478869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:23:01.134961+0000","scrub_duration":0.031767374000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"918'8","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.900114+0000","last_change":"2023-08-22T23:44:52.901399+0000","last_active":"2023-08-22T23:44:52.900114+0000","last_peered":"2023-08-22T23:44:52.900114+0000","last_clean":"2023-08-22T23:44:52.900114+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:52.900114+0000","last_undegraded":"2023-08-22T23:44:52.900114+0000","last_fullsized":"2023-08-22T23:44:52.900114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8","last_scrub_stamp":"2023-08-22T23:44:52.898702+0000","last_deep_scrub":"918'8","last_deep_scrub_stamp":"2023-08-22T23:44:52.898702+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.898702+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:50:16.475293+0000","scrub_duration":0.076325736000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:50.238325+0000","last_change":"2023-08-22T23:44:50.250441+0000","last_active":"2023-08-22T23:44:50.238325+0000","last_peered":"2023-08-22T23:44:50.238325+0000","last_clean":"2023-08-22T23:44:50.238325+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:50.238325+0000","last_undegraded":"2023-08-22T23:44:50.238325+0000","last_fullsized":"2023-08-22T23:44:50.238325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:50.229791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:50.229791+0000","last_clean_scrub_stamp":"2023-08-22T23:44:50.229791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:58:24.193218+0000","scrub_duration":0.21635689899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.008748+0000","last_change":"2023-08-22T23:44:54.009903+0000","last_active":"2023-08-22T23:44:54.008748+0000","last_peered":"2023-08-22T23:44:54.008748+0000","last_clean":"2023-08-22T23:44:54.008748+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:54.008748+0000","last_undegraded":"2023-08-22T23:44:54.008748+0000","last_fullsized":"2023-08-22T23:44:54.008748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.007311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.007311+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.007311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:49:21.630484+0000","scrub_duration":0.043985337999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:59.284695+0000","last_change":"2023-08-22T23:44:59.285871+0000","last_active":"2023-08-22T23:44:59.284695+0000","last_peered":"2023-08-22T23:44:59.284695+0000","last_clean":"2023-08-22T23:44:59.284695+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:59.284695+0000","last_undegraded":"2023-08-22T23:44:59.284695+0000","last_fullsized":"2023-08-22T23:44:59.284695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:59.283191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:59.283191+0000","last_clean_scrub_stamp":"2023-08-22T23:44:59.283191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:16:55.698349+0000","scrub_duration":0.032512915000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"918'78","reported_seq":995,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.503440+0000","last_change":"2023-08-22T23:44:51.507885+0000","last_active":"2023-08-22T23:44:51.503440+0000","last_peered":"2023-08-22T23:44:51.503440+0000","last_clean":"2023-08-22T23:44:51.503440+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:51.503440+0000","last_undegraded":"2023-08-22T23:44:51.503440+0000","last_fullsized":"2023-08-22T23:44:51.503440+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'78","last_scrub_stamp":"2023-08-22T23:44:51.501949+0000","last_deep_scrub":"918'78","last_deep_scrub_stamp":"2023-08-22T23:44:51.501949+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.501949+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:17:02.406666+0000","scrub_duration":0.125489981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:00.060950+0000","last_change":"2023-08-22T23:45:00.062115+0000","last_active":"2023-08-22T23:45:00.060950+0000","last_peered":"2023-08-22T23:45:00.060950+0000","last_clean":"2023-08-22T23:45:00.060950+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:45:00.060950+0000","last_undegraded":"2023-08-22T23:45:00.060950+0000","last_fullsized":"2023-08-22T23:45:00.060950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:00.059445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:00.059445+0000","last_clean_scrub_stamp":"2023-08-22T23:45:00.059445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:10:12.317174+0000","scrub_duration":0.060663304000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.843354+0000","last_change":"2023-08-22T23:23:33.761439+0000","last_active":"2023-08-22T23:44:39.843354+0000","last_peered":"2023-08-22T23:44:39.843354+0000","last_clean":"2023-08-22T23:44:39.843354+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:44:39.843354+0000","last_undegraded":"2023-08-22T23:44:39.843354+0000","last_fullsized":"2023-08-22T23:44:39.843354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.851266+0000","last_change":"2023-08-22T23:23:33.024596+0000","last_active":"2023-08-22T23:44:39.851266+0000","last_peered":"2023-08-22T23:44:39.851266+0000","last_clean":"2023-08-22T23:44:39.851266+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:44:39.851266+0000","last_undegraded":"2023-08-22T23:44:39.851266+0000","last_fullsized":"2023-08-22T23:44:39.851266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.573413+0000","last_change":"2023-08-22T23:44:54.574547+0000","last_active":"2023-08-22T23:44:54.573413+0000","last_peered":"2023-08-22T23:44:54.573413+0000","last_clean":"2023-08-22T23:44:54.573413+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:54.573413+0000","last_undegraded":"2023-08-22T23:44:54.573413+0000","last_fullsized":"2023-08-22T23:44:54.573413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.571987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.571987+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.571987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:24:26.605250+0000","scrub_duration":0.091127818999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:46.514602+0000","last_change":"2023-08-22T23:44:46.515850+0000","last_active":"2023-08-22T23:44:46.514602+0000","last_peered":"2023-08-22T23:44:46.514602+0000","last_clean":"2023-08-22T23:44:46.514602+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:46.514602+0000","last_undegraded":"2023-08-22T23:44:46.514602+0000","last_fullsized":"2023-08-22T23:44:46.514602+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_clean_scrub_stamp":"2023-08-22T23:44:46.513081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:49:45.949593+0000","scrub_duration":0.047647529000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.867039+0000","last_change":"2023-08-22T23:23:32.954050+0000","last_active":"2023-08-22T23:44:39.867039+0000","last_peered":"2023-08-22T23:44:39.867039+0000","last_clean":"2023-08-22T23:44:39.867039+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:44:39.867039+0000","last_undegraded":"2023-08-22T23:44:39.867039+0000","last_fullsized":"2023-08-22T23:44:39.867039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.673754+0000","last_change":"2023-08-22T23:44:52.674889+0000","last_active":"2023-08-22T23:44:52.673754+0000","last_peered":"2023-08-22T23:44:52.673754+0000","last_clean":"2023-08-22T23:44:52.673754+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:52.673754+0000","last_undegraded":"2023-08-22T23:44:52.673754+0000","last_fullsized":"2023-08-22T23:44:52.673754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.672298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.672298+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.672298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:59:07.685804+0000","scrub_duration":0.032147282999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:49.599748+0000","last_change":"2023-08-22T23:44:49.600902+0000","last_active":"2023-08-22T23:44:49.599748+0000","last_peered":"2023-08-22T23:44:49.599748+0000","last_clean":"2023-08-22T23:44:49.599748+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:49.599748+0000","last_undegraded":"2023-08-22T23:44:49.599748+0000","last_fullsized":"2023-08-22T23:44:49.599748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:49.598301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:49.598301+0000","last_clean_scrub_stamp":"2023-08-22T23:44:49.598301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:13:36.863120+0000","scrub_duration":0.084724123999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.377506+0000","last_change":"2023-08-22T23:44:53.378666+0000","last_active":"2023-08-22T23:44:53.377506+0000","last_peered":"2023-08-22T23:44:53.377506+0000","last_clean":"2023-08-22T23:44:53.377506+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:53.377506+0000","last_undegraded":"2023-08-22T23:44:53.377506+0000","last_fullsized":"2023-08-22T23:44:53.377506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.376022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.376022+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.376022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:07:11.775246+0000","scrub_duration":0.036018521999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.839654+0000","last_change":"2023-08-22T23:23:45.510476+0000","last_active":"2023-08-22T23:44:39.839654+0000","last_peered":"2023-08-22T23:44:39.839654+0000","last_clean":"2023-08-22T23:44:39.839654+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:44:39.839654+0000","last_undegraded":"2023-08-22T23:44:39.839654+0000","last_fullsized":"2023-08-22T23:44:39.839654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":869,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.896115+0000","last_change":"2023-08-22T23:44:54.897241+0000","last_active":"2023-08-22T23:44:54.896115+0000","last_peered":"2023-08-22T23:44:54.896115+0000","last_clean":"2023-08-22T23:44:54.896115+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:54.896115+0000","last_undegraded":"2023-08-22T23:44:54.896115+0000","last_fullsized":"2023-08-22T23:44:54.896115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.892207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.892207+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.892207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:22:01.531638+0000","scrub_duration":0.03351606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":883,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.983172+0000","last_change":"2023-08-22T23:23:41.098969+0000","last_active":"2023-08-22T23:44:39.983172+0000","last_peered":"2023-08-22T23:44:39.983172+0000","last_clean":"2023-08-22T23:44:39.983172+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:44:39.983172+0000","last_undegraded":"2023-08-22T23:44:39.983172+0000","last_fullsized":"2023-08-22T23:44:39.983172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477010,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65888,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305952,"statfs":{"total":96636764160,"available":96569294848,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":13.682,"5min":16.966000000000001,"15min":19.555},"min":{"1min":7.7030000000000003,"5min":4.734,"15min":4.734},"max":{"1min":40.639000000000003,"5min":187.08799999999999,"15min":254.63},"last":10.401},{"interface":"front","average":{"1min":17.207999999999998,"5min":23.309999999999999,"15min":22.024999999999999},"min":{"1min":6.1360000000000001,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":47.487000000000002,"5min":330.00099999999998,"15min":330.00099999999998},"last":9.0440000000000005}]},{"osd":2,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.364000000000001,"5min":22.524999999999999,"15min":22.077000000000002},"min":{"1min":6.8159999999999998,"5min":6.8099999999999996,"15min":6.3019999999999996},"max":{"1min":52.658000000000001,"5min":176.803,"15min":176.803},"last":12.645},{"interface":"front","average":{"1min":33.201000000000001,"5min":28.675999999999998,"15min":25.103000000000002},"min":{"1min":8.7319999999999993,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":267.399,"5min":277.048,"15min":277.048},"last":7.4740000000000002}]},{"osd":3,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.978000000000002,"5min":15.691000000000001,"15min":19.866},"min":{"1min":7.7519999999999998,"5min":6.3860000000000001,"15min":5.766},"max":{"1min":112.148,"5min":112.148,"15min":332.09199999999998},"last":8.2940000000000005},{"interface":"front","average":{"1min":18.84,"5min":24.292000000000002,"15min":25.629000000000001},"min":{"1min":7.0739999999999998,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":75.138000000000005,"5min":193.38499999999999,"15min":218.94900000000001},"last":15.484999999999999}]},{"osd":4,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":21.919,"5min":22.414999999999999,"15min":25.763999999999999},"min":{"1min":9.3780000000000001,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":107.697,"5min":107.697,"15min":333.517},"last":14.843},{"interface":"front","average":{"1min":21.629000000000001,"5min":28.734000000000002,"15min":29.236000000000001},"min":{"1min":9.7230000000000008,"5min":9.1370000000000005,"15min":8.5779999999999994},"max":{"1min":108.776,"5min":273.65800000000002,"15min":342.31999999999999},"last":14.08}]},{"osd":5,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":22.91,"5min":21.954999999999998,"15min":23.404},"min":{"1min":11.204000000000001,"5min":6.0869999999999997,"15min":6.0869999999999997},"max":{"1min":73.780000000000001,"5min":106.52,"15min":326.71300000000002},"last":16.164999999999999},{"interface":"front","average":{"1min":20.088999999999999,"5min":24.102,"15min":27.399999999999999},"min":{"1min":7.4470000000000001,"5min":7.4470000000000001,"15min":7.2830000000000004},"max":{"1min":99.808000000000007,"5min":196.352,"15min":250.43600000000001},"last":11.971}]},{"osd":6,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":18.93,"5min":23.151,"15min":26.940999999999999},"min":{"1min":10.054,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":50.091999999999999,"5min":110.843,"15min":267.84800000000001},"last":17.533000000000001},{"interface":"front","average":{"1min":24.856000000000002,"5min":30.155999999999999,"15min":29.498999999999999},"min":{"1min":12.632,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":103.387,"5min":198.358,"15min":255.58699999999999},"last":16.812000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:44:04 2023","interfaces":[{"interface":"back","average":{"1min":23.925999999999998,"5min":33.804000000000002,"15min":32.649000000000001},"min":{"1min":13.246,"5min":11.016,"15min":10.459},"max":{"1min":76.736999999999995,"5min":274.60899999999998,"15min":274.60899999999998},"last":19.204999999999998},{"interface":"front","average":{"1min":22.082000000000001,"5min":22.986999999999998,"15min":24.119},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":7.2030000000000003},"max":{"1min":98.132999999999996,"5min":114.447,"15min":213.56899999999999},"last":11.247999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477012,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83544,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94288296,"statfs":{"total":96636764160,"available":96551215104,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":15.458,"5min":16.449999999999999,"15min":18.245999999999999},"min":{"1min":6.625,"5min":5.6189999999999998,"15min":5.0359999999999996},"max":{"1min":60.25,"5min":153.98599999999999,"15min":220.95400000000001},"last":9.2680000000000007},{"interface":"front","average":{"1min":20.081,"5min":23.474,"15min":25.812000000000001},"min":{"1min":10.656000000000001,"5min":9.2579999999999991,"15min":6.3620000000000001},"max":{"1min":65.772000000000006,"5min":215.34800000000001,"15min":356.75599999999997},"last":19.198}]},{"osd":1,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":13.866,"5min":16.361000000000001,"15min":19.143000000000001},"min":{"1min":6.2240000000000002,"5min":5.218,"15min":5.141},"max":{"1min":59.353999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":11.356},{"interface":"front","average":{"1min":14.013,"5min":16.992000000000001,"15min":19.372},"min":{"1min":8.0760000000000005,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":63.573,"5min":190.49799999999999,"15min":374.01799999999997},"last":10.022}]},{"osd":2,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":17.425000000000001,"5min":22.100000000000001,"15min":21.399000000000001},"min":{"1min":8.0069999999999997,"5min":6.524,"15min":6.524},"max":{"1min":64.561000000000007,"5min":198.649,"15min":233.745},"last":10.694000000000001},{"interface":"front","average":{"1min":17.495999999999999,"5min":20.888000000000002,"15min":22.417999999999999},"min":{"1min":7.8179999999999996,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":70.597999999999999,"5min":193.53999999999999,"15min":241.23500000000001},"last":12.81}]},{"osd":4,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":17.477,"5min":21.974,"15min":27.140000000000001},"min":{"1min":8.7720000000000002,"5min":8.7720000000000002,"15min":6.6230000000000002},"max":{"1min":72.275000000000006,"5min":199.42400000000001,"15min":370.84399999999999},"last":14.433999999999999},{"interface":"front","average":{"1min":19.637,"5min":24.417000000000002,"15min":30.408999999999999},"min":{"1min":8.4450000000000003,"5min":6.726,"15min":6.726},"max":{"1min":89.596999999999994,"5min":231.53,"15min":372.45299999999997},"last":17.411000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":21.088999999999999,"15min":24.824999999999999},"min":{"1min":9.6869999999999994,"5min":9.6869999999999994,"15min":8.5419999999999998},"max":{"1min":67.260999999999996,"5min":175.429,"15min":241.86099999999999},"last":15.090999999999999},{"interface":"front","average":{"1min":18.904,"5min":20.405000000000001,"15min":24.536000000000001},"min":{"1min":8.6530000000000005,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":74.754999999999995,"5min":212.09,"15min":289.34300000000002},"last":12.169}]},{"osd":6,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":20.495000000000001,"5min":22.859999999999999,"15min":27.015999999999998},"min":{"1min":11.388999999999999,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":68.186000000000007,"5min":213.15000000000001,"15min":246.30000000000001},"last":18.318999999999999},{"interface":"front","average":{"1min":25.173999999999999,"5min":28.638000000000002,"15min":27.481999999999999},"min":{"1min":12.978,"5min":8.1669999999999998,"15min":7.7830000000000004},"max":{"1min":73.796000000000006,"5min":215.41300000000001,"15min":215.41300000000001},"last":20.613}]},{"osd":7,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":22.321999999999999,"5min":22.753,"15min":28.486000000000001},"min":{"1min":12.140000000000001,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":76.263999999999996,"5min":206.708,"15min":417.863},"last":13.492000000000001},{"interface":"front","average":{"1min":22.802,"5min":27.838999999999999,"15min":28.358000000000001},"min":{"1min":13.613,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":71.433999999999997,"5min":229.85900000000001,"15min":229.85900000000001},"last":21.623000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477010,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65500,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94306340,"statfs":{"total":96636764160,"available":96569692160,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":8.4800000000000004,"5min":16.053999999999998,"15min":17.029},"min":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":15.577999999999999,"5min":204.17500000000001,"15min":208.114},"last":14.943},{"interface":"front","average":{"1min":16.081,"5min":21.925999999999998,"15min":20.050000000000001},"min":{"1min":8.7599999999999998,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":44.661000000000001,"5min":205.833,"15min":205.833},"last":11.52}]},{"osd":1,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":19.687999999999999,"5min":17.501000000000001,"15min":18.324999999999999},"min":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":114.821,"5min":142.42599999999999,"15min":218.06200000000001},"last":10.829000000000001},{"interface":"front","average":{"1min":24.239999999999998,"5min":22.905000000000001,"15min":21.925999999999998},"min":{"1min":10.121,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":115.809,"5min":145.83000000000001,"15min":230.108},"last":17.841999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:44:54 2023","interfaces":[{"interface":"back","average":{"1min":11.27,"5min":21.934999999999999,"15min":20.477},"min":{"1min":6.9749999999999996,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":23.353999999999999,"5min":292.49000000000001,"15min":292.49000000000001},"last":10.287000000000001},{"interface":"front","average":{"1min":20.030000000000001,"5min":28.835999999999999,"15min":25.010999999999999},"min":{"1min":10.519,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":65.405000000000001,"5min":304.017,"15min":306.26299999999998},"last":22.596}]},{"osd":4,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":15.987,"5min":19.297999999999998,"15min":21.773},"min":{"1min":8.7170000000000005,"5min":8.3759999999999994,"15min":7.3869999999999996},"max":{"1min":52.591999999999999,"5min":157.559,"15min":187.357},"last":13.217000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":18.696999999999999,"15min":20.98},"min":{"1min":7.7050000000000001,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":19.834,"5min":146.64599999999999,"15min":206.59100000000001},"last":18.574000000000002}]},{"osd":5,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.920000000000002,"5min":22.468,"15min":24.614000000000001},"min":{"1min":10.135,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":54.234000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":20.995999999999999},{"interface":"front","average":{"1min":22.023,"5min":21.733000000000001,"15min":23.268000000000001},"min":{"1min":8.9309999999999992,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":63.061999999999998,"5min":181.24199999999999,"15min":181.24199999999999},"last":16.065999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":27.670999999999999,"5min":25.062999999999999,"15min":24.218},"min":{"1min":11.868,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":127.512,"5min":189.55000000000001,"15min":189.55000000000001},"last":23.396999999999998},{"interface":"front","average":{"1min":25.172999999999998,"5min":21.625,"15min":24.170000000000002},"min":{"1min":9.8800000000000008,"5min":9.0280000000000005,"15min":8.8819999999999997},"max":{"1min":119.714,"5min":129.928,"15min":187.434},"last":19.373999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:44:15 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":26.472999999999999,"15min":29.672000000000001},"min":{"1min":11.529999999999999,"5min":10.513,"15min":9.9939999999999998},"max":{"1min":55.665999999999997,"5min":195.655,"15min":402.06099999999998},"last":25.468},{"interface":"front","average":{"1min":17.021000000000001,"5min":21.558,"15min":28.571999999999999},"min":{"1min":8.9019999999999992,"5min":8.6690000000000005,"15min":8.6690000000000005},"max":{"1min":39.817,"5min":158.26499999999999,"15min":427.274},"last":16.782}]}]},{"osd":0,"up_from":16,"seq":68719477012,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66152,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305688,"statfs":{"total":96636764160,"available":96569024512,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.164,"15min":17.242999999999999},"min":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":28.050000000000001,"5min":125.212,"15min":315.35300000000001},"last":9.7010000000000005},{"interface":"front","average":{"1min":11.481999999999999,"5min":19.741,"15min":19.794},"min":{"1min":5.8739999999999997,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":27.388000000000002,"5min":150.68199999999999,"15min":314.375},"last":17.478999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":15.347,"5min":22.693000000000001,"15min":21.102},"min":{"1min":7.5519999999999996,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":42.912999999999997,"5min":133.678,"15min":255.041},"last":10.380000000000001},{"interface":"front","average":{"1min":14.930999999999999,"5min":23.047000000000001,"15min":21.428000000000001},"min":{"1min":7.1859999999999999,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":31.559999999999999,"5min":124.077,"15min":266.08600000000001},"last":19.626999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":14.122,"5min":21.184999999999999,"15min":20.628},"min":{"1min":7.6429999999999998,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":77.963999999999999,"5min":140.357,"15min":359.37},"last":8.4269999999999996},{"interface":"front","average":{"1min":20.382000000000001,"5min":23.077999999999999,"15min":22.991},"min":{"1min":8.3200000000000003,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":80.066999999999993,"5min":133.27000000000001,"15min":335.35399999999998},"last":18.414999999999999}]},{"osd":4,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":17.265999999999998,"5min":21.844999999999999,"15min":22.927},"min":{"1min":9.0359999999999996,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":40.703000000000003,"5min":270.06900000000002,"15min":270.06900000000002},"last":14.631},{"interface":"front","average":{"1min":17.626999999999999,"5min":23.300000000000001,"15min":23.582000000000001},"min":{"1min":7.2880000000000003,"5min":7.1689999999999996,"15min":7.1689999999999996},"max":{"1min":48.825000000000003,"5min":204.54499999999999,"15min":301.78399999999999},"last":11.739000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:45:04 2023","interfaces":[{"interface":"back","average":{"1min":27.023,"5min":25.725999999999999,"15min":25.164000000000001},"min":{"1min":8.9079999999999995,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":263.81599999999997,"5min":263.81599999999997,"15min":287.80200000000002},"last":12.526},{"interface":"front","average":{"1min":29.484999999999999,"5min":25.931000000000001,"15min":26.088000000000001},"min":{"1min":10.371,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":272.935,"5min":272.935,"15min":272.935},"last":20.559000000000001}]},{"osd":6,"last update":"Tue Aug 22 23:44:22 2023","interfaces":[{"interface":"back","average":{"1min":25.445,"5min":28.920999999999999,"15min":25.911000000000001},"min":{"1min":10.363,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":159.46299999999999,"5min":243.15100000000001,"15min":276.935},"last":21.471},{"interface":"front","average":{"1min":21.541,"5min":23.311,"15min":24.834},"min":{"1min":10.583,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":109.886,"5min":241.351,"15min":265.17399999999998},"last":13.93}]},{"osd":7,"last update":"Tue Aug 22 23:45:04 2023","interfaces":[{"interface":"back","average":{"1min":18.571000000000002,"5min":27.088999999999999,"15min":30.423999999999999},"min":{"1min":14.311999999999999,"5min":10.805999999999999,"15min":7.1760000000000002},"max":{"1min":32.917000000000002,"5min":184.333,"15min":770.02200000000005},"last":15.363},{"interface":"front","average":{"1min":15.192,"5min":24.359999999999999,"15min":30.084},"min":{"1min":10.214,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":21.288,"5min":154.00800000000001,"15min":834.71799999999996},"last":13.185}]}]},{"osd":6,"up_from":15,"seq":64424509716,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65500,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94306340,"statfs":{"total":96636764160,"available":96569692160,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":15.823,"5min":15.853999999999999,"15min":16.640999999999998},"min":{"1min":5.774,"5min":5.774,"15min":4.5570000000000004},"max":{"1min":77.439999999999998,"5min":101.271,"15min":190.38800000000001},"last":14.834},{"interface":"front","average":{"1min":25.606000000000002,"5min":23.318999999999999,"15min":21.681000000000001},"min":{"1min":9.0399999999999991,"5min":9.0399999999999991,"15min":4.9699999999999998},"max":{"1min":120.398,"5min":122.059,"15min":215.38800000000001},"last":17.649000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":18.706,"5min":15.819000000000001,"15min":16.527999999999999},"min":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.1100000000000003},"max":{"1min":46.259999999999998,"5min":77.727999999999994,"15min":222.346},"last":8.2300000000000004},{"interface":"front","average":{"1min":21.076000000000001,"5min":18.771999999999998,"15min":19.385000000000002},"min":{"1min":7.4039999999999999,"5min":7.4039999999999999,"15min":6.0519999999999996},"max":{"1min":53.518000000000001,"5min":137.86799999999999,"15min":216.02000000000001},"last":15.488}]},{"osd":2,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":21.442,"5min":18.684000000000001,"15min":19.425000000000001},"min":{"1min":6.6619999999999999,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":92.921000000000006,"5min":171.38399999999999,"15min":283.65600000000001},"last":9.125},{"interface":"front","average":{"1min":22.297999999999998,"5min":21.972000000000001,"15min":23.984999999999999},"min":{"1min":9.0169999999999995,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":100.437,"5min":170.02600000000001,"15min":389.19299999999998},"last":14.063000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":22.475999999999999,"15min":27.465},"min":{"1min":7.6630000000000003,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":208.94800000000001,"5min":214.666,"15min":352.79500000000002},"last":9.9019999999999992},{"interface":"front","average":{"1min":25.527000000000001,"5min":23.891999999999999,"15min":24.518000000000001},"min":{"1min":10.247999999999999,"5min":7.4370000000000003,"15min":7.415},"max":{"1min":91.753,"5min":178.21600000000001,"15min":314.69},"last":18.494}]},{"osd":4,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":16.306999999999999,"5min":16.852,"15min":19.088999999999999},"min":{"1min":8.1959999999999997,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":40.009999999999998,"5min":91.307000000000002,"15min":151.41399999999999},"last":6.8550000000000004},{"interface":"front","average":{"1min":22.344000000000001,"5min":22.126000000000001,"15min":26.108000000000001},"min":{"1min":10.922000000000001,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":50.005000000000003,"5min":92.557000000000002,"15min":156.40100000000001},"last":11.888999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":22.686,"5min":24.760999999999999,"15min":26.306000000000001},"min":{"1min":10.253,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":86.361999999999995,"5min":183.268,"15min":249.37299999999999},"last":16.774000000000001},{"interface":"front","average":{"1min":24.760000000000002,"5min":22.363,"15min":23.788},"min":{"1min":9.9090000000000007,"5min":9.2989999999999995,"15min":7.3849999999999998},"max":{"1min":93.683000000000007,"5min":146.96199999999999,"15min":215.76499999999999},"last":10.615}]},{"osd":7,"last update":"Tue Aug 22 23:44:21 2023","interfaces":[{"interface":"back","average":{"1min":19.669,"5min":22.631,"15min":26.940000000000001},"min":{"1min":10.702,"5min":10.388999999999999,"15min":9.3559999999999999},"max":{"1min":46.847999999999999,"5min":135.65700000000001,"15min":457.08199999999999},"last":11.246},{"interface":"front","average":{"1min":19.600999999999999,"5min":26.585999999999999,"15min":26.331},"min":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":54.695,"5min":247.423,"15min":353.75900000000001},"last":19.353000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509715,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88296,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283544,"statfs":{"total":96636764160,"available":96546349056,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":20.602,"5min":22.466999999999999,"15min":22.718},"min":{"1min":4.7880000000000003,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":90.212000000000003,"5min":141.732,"15min":235.06100000000001},"last":9.3079999999999998},{"interface":"front","average":{"1min":25.978000000000002,"5min":24.094000000000001,"15min":27.666},"min":{"1min":7.5999999999999996,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":112.598,"5min":189.517,"15min":395.15300000000002},"last":7.6059999999999999}]},{"osd":1,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":36.859999999999999,"5min":25.800999999999998,"15min":26.106999999999999},"min":{"1min":6.9509999999999996,"5min":6.7149999999999999,"15min":5.7439999999999998},"max":{"1min":228.94499999999999,"5min":241.87799999999999,"15min":382.04199999999997},"last":10.099},{"interface":"front","average":{"1min":41.487000000000002,"5min":33.701999999999998,"15min":33.277999999999999},"min":{"1min":11.436999999999999,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":237.38499999999999,"5min":252.441,"15min":382.69},"last":13.920999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":23.855,"5min":24.207999999999998,"15min":25.460999999999999},"min":{"1min":6.2359999999999998,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":214.36500000000001,"5min":219.37899999999999,"15min":537.76199999999994},"last":7.0289999999999999},{"interface":"front","average":{"1min":20.466999999999999,"5min":26.917999999999999,"15min":32.433999999999997},"min":{"1min":6.5999999999999996,"5min":6.5999999999999996,"15min":6.1210000000000004},"max":{"1min":140.50200000000001,"5min":214.50700000000001,"15min":556.82899999999995},"last":16.753}]},{"osd":3,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":24.536999999999999,"5min":24.959,"15min":28.465},"min":{"1min":8.3970000000000002,"5min":7.0449999999999999,"15min":6.6159999999999997},"max":{"1min":160.98400000000001,"5min":161.05199999999999,"15min":354.50400000000002},"last":10.859},{"interface":"front","average":{"1min":31.042999999999999,"5min":30.719999999999999,"15min":30.981999999999999},"min":{"1min":10.749000000000001,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":226.25700000000001,"5min":226.25700000000001,"15min":339.51499999999999},"last":13.263}]},{"osd":5,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":34.338000000000001,"5min":30.631,"15min":32.762999999999998},"min":{"1min":9.1660000000000004,"5min":7.4160000000000004,"15min":7.2279999999999998},"max":{"1min":139.31899999999999,"5min":238.57300000000001,"15min":380.863},"last":14.696},{"interface":"front","average":{"1min":26.352,"5min":30.120999999999999,"15min":36.542000000000002},"min":{"1min":7.5999999999999996,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":99.632999999999996,"5min":215.928,"15min":364.09500000000003},"last":11.483000000000001}]},{"osd":6,"last update":"Tue Aug 22 23:44:10 2023","interfaces":[{"interface":"back","average":{"1min":32.222000000000001,"5min":30.013999999999999,"15min":31.196000000000002},"min":{"1min":12.083,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":190.446,"5min":217,"15min":249.29599999999999},"last":15.326000000000001},{"interface":"front","average":{"1min":24.103000000000002,"5min":24.334,"15min":31.428000000000001},"min":{"1min":8.952,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":132.864,"5min":220.87100000000001,"15min":468.887},"last":17.603000000000002}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.209,"5min":24.561,"15min":30.809000000000001},"min":{"1min":10.443,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":26.222999999999999,"5min":187.30099999999999,"15min":361.45499999999998},"last":18.454000000000001},{"interface":"front","average":{"1min":19.317,"5min":28.436,"15min":31.718},"min":{"1min":14.016999999999999,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":33.298999999999999,"5min":197.05600000000001,"15min":368.08699999999999},"last":19.399000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542421,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87640,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94284200,"statfs":{"total":96636764160,"available":96547020800,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":11.916,"5min":19.649000000000001,"15min":20.076000000000001},"min":{"1min":5.6710000000000003,"5min":5.6710000000000003,"15min":4.3040000000000003},"max":{"1min":38.225999999999999,"5min":307.584,"15min":307.584},"last":17.436},{"interface":"front","average":{"1min":16.166,"5min":21.260999999999999,"15min":20.367000000000001},"min":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":48.941000000000003,"5min":317.81,"15min":317.81},"last":18.120000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":13.314,"5min":17.096,"15min":25.003},"min":{"1min":6.2629999999999999,"5min":5.077,"15min":5.077},"max":{"1min":22.173999999999999,"5min":159.58799999999999,"15min":319.34500000000003},"last":16.718},{"interface":"front","average":{"1min":11.484999999999999,"5min":14.808,"15min":20.055},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":26.338000000000001,"5min":160.74600000000001,"15min":195.78999999999999},"last":10.454000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":15.140000000000001,"5min":17.152999999999999,"15min":23.173999999999999},"min":{"1min":8.6620000000000008,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":47.432000000000002,"5min":148.97800000000001,"15min":772.00400000000002},"last":18.818000000000001},{"interface":"front","average":{"1min":17.105,"5min":20.027999999999999,"15min":24.41},"min":{"1min":10.882,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":43.933999999999997,"5min":151.46700000000001,"15min":789.553},"last":7.4160000000000004}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.602,"5min":19.146000000000001,"15min":22.295999999999999},"min":{"1min":7.7370000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":70.665999999999997,"5min":154.44200000000001,"15min":243.33699999999999},"last":11.148999999999999},{"interface":"front","average":{"1min":18.843,"5min":24.088999999999999,"15min":26.478000000000002},"min":{"1min":6.7880000000000003,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":68.334000000000003,"5min":337.09699999999998,"15min":337.09699999999998},"last":20.254999999999999}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":20.622,"5min":21.59,"15min":29.172999999999998},"min":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.0570000000000004},"max":{"1min":109.88800000000001,"5min":111.015,"15min":333.04000000000002},"last":8.1590000000000007},{"interface":"front","average":{"1min":18.895,"5min":20.015999999999998,"15min":24.294},"min":{"1min":8.7140000000000004,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":97.224999999999994,"5min":117.06100000000001,"15min":240.084},"last":11.814}]},{"osd":5,"last update":"Tue Aug 22 23:45:06 2023","interfaces":[{"interface":"back","average":{"1min":15.492000000000001,"5min":20.783000000000001,"15min":28.716999999999999},"min":{"1min":11.125999999999999,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":23.506,"5min":247.898,"15min":342.53100000000001},"last":12.635},{"interface":"front","average":{"1min":18.222000000000001,"5min":21.045000000000002,"15min":26.207999999999998},"min":{"1min":10.242000000000001,"5min":8.8770000000000007,"15min":8.5809999999999995},"max":{"1min":33.665999999999997,"5min":135.31399999999999,"15min":271.47500000000002},"last":14.59}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.318000000000001,"5min":21.977,"15min":25.302},"min":{"1min":8.5459999999999994,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":32.176000000000002,"5min":144.21600000000001,"15min":228.43100000000001},"last":15.571999999999999},{"interface":"front","average":{"1min":17.408000000000001,"5min":23.198,"15min":28.206},"min":{"1min":10.394,"5min":10.090999999999999,"15min":9.2650000000000006},"max":{"1min":24.224,"5min":141.92599999999999,"15min":262.45800000000003},"last":19.556999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542418,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83936,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287904,"statfs":{"total":96636764160,"available":96550813696,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":12.601000000000001,"5min":16.588000000000001,"15min":20.968},"min":{"1min":6.7779999999999996,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":33.573999999999998,"5min":114.51300000000001,"15min":220.017},"last":7.6500000000000004},{"interface":"front","average":{"1min":18.562000000000001,"5min":22.257999999999999,"15min":27.212},"min":{"1min":10.353999999999999,"5min":6.7919999999999998,"15min":5.9329999999999998},"max":{"1min":32.883000000000003,"5min":115.74299999999999,"15min":264.11900000000003},"last":8.5389999999999997}]},{"osd":1,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":10.41,"5min":14.409000000000001,"15min":20.576000000000001},"min":{"1min":6.5119999999999996,"5min":6.1289999999999996,"15min":5.9379999999999997},"max":{"1min":16.731000000000002,"5min":102.66200000000001,"15min":343.899},"last":17.314},{"interface":"front","average":{"1min":15.853,"5min":21.071999999999999,"15min":25.216000000000001},"min":{"1min":8.7690000000000001,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":27.713000000000001,"5min":111.873,"15min":270.40499999999997},"last":10.125}]},{"osd":2,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":14.057,"5min":17.219000000000001,"15min":21.003},"min":{"1min":6.4080000000000004,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":28.347000000000001,"5min":116.45,"15min":248.74799999999999},"last":16.303999999999998},{"interface":"front","average":{"1min":14.875,"5min":17.582000000000001,"15min":24.184999999999999},"min":{"1min":7.0060000000000002,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":26.632000000000001,"5min":114.843,"15min":256.94400000000002},"last":15.131}]},{"osd":3,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":24.103999999999999,"15min":30.831},"min":{"1min":7.6070000000000002,"5min":6.5739999999999998,"15min":6.5739999999999998},"max":{"1min":50.060000000000002,"5min":231.99799999999999,"15min":257.23500000000001},"last":18.030000000000001},{"interface":"front","average":{"1min":25.657,"5min":23.934999999999999,"15min":24.399000000000001},"min":{"1min":9.8130000000000006,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":62.527999999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":11.885999999999999}]},{"osd":4,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":17.437999999999999,"5min":25.553000000000001,"15min":28.288},"min":{"1min":9.5950000000000006,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":33.143000000000001,"5min":208.03200000000001,"15min":374.83499999999998},"last":20.058},{"interface":"front","average":{"1min":15.750999999999999,"5min":25.452999999999999,"15min":26.808},"min":{"1min":9.1219999999999999,"5min":7.5389999999999997,"15min":6.3019999999999996},"max":{"1min":22.956,"5min":273.58699999999999,"15min":341.255},"last":19.245999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":21.895,"5min":26.039000000000001,"15min":31.167999999999999},"min":{"1min":9.6560000000000006,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":49.250999999999998,"5min":228.626,"15min":241.26499999999999},"last":12.584},{"interface":"front","average":{"1min":16.486000000000001,"5min":23.117000000000001,"15min":26.573},"min":{"1min":8.891,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":32.026000000000003,"5min":247.06800000000001,"15min":247.06800000000001},"last":22.311}]},{"osd":7,"last update":"Tue Aug 22 23:44:16 2023","interfaces":[{"interface":"back","average":{"1min":19.277999999999999,"5min":21.434000000000001,"15min":27.062999999999999},"min":{"1min":10.872,"5min":8.2560000000000002,"15min":8.2560000000000002},"max":{"1min":31.568000000000001,"5min":121.524,"15min":317.44600000000003},"last":23.068999999999999},{"interface":"front","average":{"1min":17.907,"5min":25.620000000000001,"15min":28.559999999999999},"min":{"1min":11.589,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":30.829000000000001,"5min":201.81700000000001,"15min":361.93000000000001},"last":20.902999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:45:12.122 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.123 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.123 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.124 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.124 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.124 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.125 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:12.125 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-08-22T23:45:13.569 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:58.853 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.571 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:58.855 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.588 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:23:58.872 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.620 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:58.917 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.621 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:58.919 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.622 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:58.906 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.625 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:58.909 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.636 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:58.933 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:13.639 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:23:58.923 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.597 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:08.881 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.599 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:08.883 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.612 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:08.895 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.650 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:08.933 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.650 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:08.934 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.651 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:08.948 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.652 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:08.949 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.661 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:08.945 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:23.663 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:08.961 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:32.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.238+0000 7fe75abbc640 10 monclient: get_monmap_and_config 2023-08-22T23:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.238+0000 7fe75abbc640 10 monclient: build_initial_monmap 2023-08-22T23:45:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.238+0000 7fe75abbc640 10 monclient: monmap: 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:45:32.238679+0000 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:45:32.238679+0000 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:32.239 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:45:32.240 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:45:32.241 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 1 Processor -- start 2023-08-22T23:45:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 1 -- start start 2023-08-22T23:45:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient: init 2023-08-22T23:45:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-08-22T23:45:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe75413e860 addr v1:172.21.15.29:6789/0 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe754144710 addr v1:172.21.15.52:6789/0 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75414aaa0 0x7fe75414ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe75414aaa0 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7540a7ce0 con 0x7fe75413e860 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7540a7f80 con 0x7fe754144710 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540a95a0 con 0x7fe75414aaa0 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.239+0000 7fe75abbc640 10 monclient(hunting): authenticate will time out at 2896.489276s 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe758958640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe75413e860 0x7fe7540a74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34982/0 (socket says 172.21.15.29:34982) 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe758958640 1 -- 172.21.15.29:0/4178682365 learned_addr learned my addr 172.21.15.29:0/4178682365 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe759159640 1 --2- 172.21.15.29:0/4178682365 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75414aaa0 0x7fe75414ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe759159640 10 monclient(hunting): get_auth_request con 0x7fe75414aaa0 auth_method 0 2023-08-22T23:45:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe759159640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe759159640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.240+0000 7fe759159640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.241+0000 7fe759159640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.241+0000 7fe759159640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.241+0000 7fe759159640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.242+0000 7fe7537fe640 1 -- 172.21.15.29:0/4178682365 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119055959 0 0) 0x7fe7540a7ce0 con 0x7fe75413e860 2023-08-22T23:45:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.242+0000 7fe7537fe640 10 monclient(hunting): _negotiate 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.242+0000 7fe7537fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.242+0000 7fe7537fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.242+0000 7fe7537fe640 10 monclient(hunting): my global_id is 14844 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.242+0000 7fe7537fe640 1 -- 172.21.15.29:0/4178682365 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73c003520 con 0x7fe75413e860 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 10 monclient(hunting): handle_auth_done global_id 16745 payload 274 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 1 -- 172.21.15.29:0/4178682365 >> v1:172.21.15.52:6789/0 conn(0x7fe754144710 legacy=0x7fe754144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 1 -- 172.21.15.29:0/4178682365 >> v1:172.21.15.29:6789/0 conn(0x7fe75413e860 legacy=0x7fe7540a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 10 monclient: _finish_hunting 0 2023-08-22T23:45:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 1 monclient: found mon.noname-b 2023-08-22T23:45:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:45:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.243+0000 7fe759159640 1 -- 172.21.15.29:0/4178682365 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540a9880 con 0x7fe75414aaa0 2023-08-22T23:45:32.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.245+0000 7fe759159640 1 --2- 172.21.15.29:0/4178682365 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75414aaa0 0x7fe75414ce90 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009dc0 tx=0x7fe74c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e4be80439acd689 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 1 -- 172.21.15.29:0/4178682365 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c02cbb0 con 0x7fe75414aaa0 2023-08-22T23:45:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:45:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: dump: 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: _finish_auth 0 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: _check_auth_tickets 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 1 -- 172.21.15.29:0/4178682365 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c02cd10 con 0x7fe75414aaa0 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe75abbc640 5 monclient: authenticate success, global_id 16745 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe75abbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe75abbc640 10 monclient: get_monmap_and_config success 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe75abbc640 4 set_mon_vals no callback set 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 1 -- 172.21.15.29:0/4178682365 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c034660 con 0x7fe75414aaa0 2023-08-22T23:45:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.248+0000 7fe7537fe640 10 monclient: dump: 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 10 monclient: shutdown 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 -- 172.21.15.29:0/4178682365 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75414aaa0 msgr2=0x7fe75414ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 --2- 172.21.15.29:0/4178682365 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75414aaa0 0x7fe75414ce90 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009dc0 tx=0x7fe74c02be00 comp rx=0 tx=0).stop 2023-08-22T23:45:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 -- 172.21.15.29:0/4178682365 shutdown_connections 2023-08-22T23:45:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 --2- 172.21.15.29:0/4178682365 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75414aaa0 0x7fe75414ce90 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 -- 172.21.15.29:0/4178682365 >> 172.21.15.29:0/4178682365 conn(0x7fe754065f10 msgr2=0x7fe7540b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 -- 172.21.15.29:0/4178682365 shutdown_connections 2023-08-22T23:45:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.249+0000 7fe75abbc640 1 -- 172.21.15.29:0/4178682365 wait complete. 2023-08-22T23:45:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.250+0000 7fe75abbc640 10 monclient: build_initial_monmap 2023-08-22T23:45:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.250+0000 7fe75abbc640 10 monclient: monmap: 2023-08-22T23:45:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:45:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:45:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:45:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.250+0000 7fe75abbc640 1 librados: starting msgr at 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.250+0000 7fe75abbc640 1 librados: starting objecter 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.250+0000 7fe75abbc640 1 Processor -- start 2023-08-22T23:45:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 -- start start 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 librados: setting wanted keys 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 librados: calling monclient init 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient: init 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75413e860 0x7fe754149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): picked mon.noname-c con 0x7fe75413e860 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe754144710 0x7fe754143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe754144710 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 0x7fe7540aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): picked mon.noname-a con 0x7fe75414aaa0 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540aa8f0 con 0x7fe75414aaa0 2023-08-22T23:45:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540b5e60 con 0x7fe754144710 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540aa550 con 0x7fe75413e860 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.251+0000 7fe75abbc640 10 monclient(hunting): authenticate will time out at 2896.501262s 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 0x7fe7540aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe758958640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75413e860 0x7fe754149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 0x7fe7540aad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39006/0 (socket says 172.21.15.29:39006) 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 1 -- 172.21.15.29:0/1003500173 learned_addr learned my addr 172.21.15.29:0/1003500173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 10 monclient(hunting): get_auth_request con 0x7fe75414aaa0 auth_method 0 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe759159640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe753fff640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe754144710 0x7fe754143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe758958640 10 monclient(hunting): get_auth_request con 0x7fe75413e860 auth_method 0 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe758958640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe758958640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe758958640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe753fff640 10 monclient(hunting): get_auth_request con 0x7fe754144710 auth_method 0 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe753fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.252+0000 7fe753fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe759159640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe759159640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe759159640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe758958640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe758958640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe758958640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.253+0000 7fe753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 10 monclient(hunting): handle_auth_done global_id 14847 payload 824 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 1 -- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75413e860 msgr2=0x7fe754149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75413e860 0x7fe754149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 1 -- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe754144710 msgr2=0x7fe754143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe754144710 0x7fe754143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 10 monclient: _finish_hunting 0 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 1 monclient: found mon.noname-a 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe759159640 1 -- 172.21.15.29:0/1003500173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c009a70 con 0x7fe75414aaa0 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.256+0000 7fe758958640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75413e860 0x7fe754149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-22T23:45:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.258+0000 7fe759159640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 0x7fe7540aad30 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009ee0 tx=0x7fe74c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.258+0000 7fe7517fa640 10 client.?.objecter ms_handle_connect 0x7fe75414aaa0 2023-08-22T23:45:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.258+0000 7fe7517fa640 10 client.?.objecter resend_mon_ops 2023-08-22T23:45:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c03e610 con 0x7fe75414aaa0 2023-08-22T23:45:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:45:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: dump: 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: _finish_auth 0 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: _check_auth_tickets 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:32.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c03e770 con 0x7fe75414aaa0 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c03eb20 con 0x7fe75414aaa0 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 5 monclient: authenticate success, global_id 14847 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 10 monclient: _renew_subs 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7540b6960 con 0x7fe75414aaa0 2023-08-22T23:45:32.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe752ffd640 4 set_mon_vals no callback set 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe7517fa640 10 monclient: dump: 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:32.275 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 10 client.14847.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:45:32.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 10 monclient: _renew_subs 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7540ad030 con 0x7fe75414aaa0 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.262+0000 7fe75abbc640 1 librados: init done 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.263+0000 7fe75abbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.263+0000 7fe75abbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.263+0000 7fe75abbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:32.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.263+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7540aaa30 con 0x7fe75414aaa0 2023-08-22T23:45:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.521+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe74c03b050 con 0x7fe75414aaa0 2023-08-22T23:45:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.521+0000 7fe7517fa640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe74006ba60 0x7fe74006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.521+0000 7fe758958640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe74006ba60 0x7fe74006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.521+0000 7fe758958640 10 monclient: get_auth_request con 0x7fe74006ba60 auth_method 0 2023-08-22T23:45:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.522+0000 7fe758958640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe74006ba60 0x7fe74006df20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe7440010e0 tx=0x7fe744007be0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.522+0000 7fe7517fa640 10 client.14847.objecter ms_handle_connect 0x7fe74006ba60 2023-08-22T23:45:32.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 744..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fe74c07dd90 con 0x7fe75414aaa0 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 10 client.14847.objecter ms_dispatch 0x7fe754000b90 osd_map(918..918 src has 744..918) v4 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 3 client.14847.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 3 client.14847.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 20 client.14847.objecter dump_active .. 0 homeless 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fe74c0763c0 con 0x7fe75414aaa0 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:32.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.548+0000 7fe7517fa640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:45:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.741+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7540a9e40 con 0x7fe74006ba60 2023-08-22T23:45:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.750+0000 7fe7517fa640 1 -- 172.21.15.29:0/1003500173 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152092 (secure 0 0 0) 0x7fe744001c60 con 0x7fe74006ba60 2023-08-22T23:45:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:45:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:45:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 10 librados: watch_flush enter 2023-08-22T23:45:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 10 librados: watch_flush exit 2023-08-22T23:45:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 20 client.14847.objecter shutdown clearing up homeless session... 2023-08-22T23:45:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 10 client.14847.objecter successfully canceled tick 2023-08-22T23:45:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe74006ba60 msgr2=0x7fe74006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe74006ba60 0x7fe74006df20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe7440010e0 tx=0x7fe744007be0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 10 monclient: shutdown 2023-08-22T23:45:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 msgr2=0x7fe7540aad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 0x7fe7540aad30 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009ee0 tx=0x7fe74c004a40 comp rx=0 tx=0).stop 2023-08-22T23:45:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 shutdown_connections 2023-08-22T23:45:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fe74006ba60 0x7fe74006df20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75414aaa0 0x7fe7540aad30 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe754144710 0x7fe754143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 --2- 172.21.15.29:0/1003500173 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fe75413e860 0x7fe754149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 >> 172.21.15.29:0/1003500173 conn(0x7fe754065f10 msgr2=0x7fe75413dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 shutdown_connections 2023-08-22T23:45:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 -- 172.21.15.29:0/1003500173 wait complete. 2023-08-22T23:45:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:32.753+0000 7fe75abbc640 1 librados: shutdown 2023-08-22T23:45:32.765 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1619,"stamp":"2023-08-22T23:45:32.585751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606456,"kb_used_data":100024,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754368264,"statfs":{"total":773094113280,"available":772473102336,"internally_reserved":0,"allocated":102424576,"data_stored":62259580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002647"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:56.865960+0000","last_change":"2023-08-22T23:44:56.867424+0000","last_active":"2023-08-22T23:44:56.865960+0000","last_peered":"2023-08-22T23:44:56.865960+0000","last_clean":"2023-08-22T23:44:56.865960+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:56.865960+0000","last_undegraded":"2023-08-22T23:44:56.865960+0000","last_fullsized":"2023-08-22T23:44:56.865960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:56.864518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:56.864518+0000","last_clean_scrub_stamp":"2023-08-22T23:44:56.864518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:47:28.870179+0000","scrub_duration":0.075777624000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:06.861939+0000","last_change":"2023-08-22T23:45:06.865880+0000","last_active":"2023-08-22T23:45:06.861939+0000","last_peered":"2023-08-22T23:45:06.861939+0000","last_clean":"2023-08-22T23:45:06.861939+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:45:06.861939+0000","last_undegraded":"2023-08-22T23:45:06.861939+0000","last_fullsized":"2023-08-22T23:45:06.861939+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:06.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:06.860450+0000","last_clean_scrub_stamp":"2023-08-22T23:45:06.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:05:25.109783+0000","scrub_duration":0.052525585999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":882,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:39.857092+0000","last_change":"2023-08-22T23:23:41.258175+0000","last_active":"2023-08-22T23:44:39.857092+0000","last_peered":"2023-08-22T23:44:39.857092+0000","last_clean":"2023-08-22T23:44:39.857092+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:44:39.857092+0000","last_undegraded":"2023-08-22T23:44:39.857092+0000","last_fullsized":"2023-08-22T23:44:39.857092+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:23:27.428724+0000","last_clean_scrub_stamp":"2023-08-22T23:23:27.428724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.370589+0000","last_change":"2023-08-22T23:44:58.375301+0000","last_active":"2023-08-22T23:44:58.370589+0000","last_peered":"2023-08-22T23:44:58.370589+0000","last_clean":"2023-08-22T23:44:58.370589+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:58.370589+0000","last_undegraded":"2023-08-22T23:44:58.370589+0000","last_fullsized":"2023-08-22T23:44:58.370589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.369087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.369087+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.369087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:24:12.816396+0000","scrub_duration":0.031867338000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:01.069233+0000","last_change":"2023-08-22T23:45:01.070353+0000","last_active":"2023-08-22T23:45:01.069233+0000","last_peered":"2023-08-22T23:45:01.069233+0000","last_clean":"2023-08-22T23:45:01.069233+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:45:01.069233+0000","last_undegraded":"2023-08-22T23:45:01.069233+0000","last_fullsized":"2023-08-22T23:45:01.069233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:01.067805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:01.067805+0000","last_clean_scrub_stamp":"2023-08-22T23:45:01.067805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:57:44.262712+0000","scrub_duration":0.080561184999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:57.348370+0000","last_change":"2023-08-22T23:44:57.349504+0000","last_active":"2023-08-22T23:44:57.348370+0000","last_peered":"2023-08-22T23:44:57.348370+0000","last_clean":"2023-08-22T23:44:57.348370+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:57.348370+0000","last_undegraded":"2023-08-22T23:44:57.348370+0000","last_fullsized":"2023-08-22T23:44:57.348370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:57.346862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:57.346862+0000","last_clean_scrub_stamp":"2023-08-22T23:44:57.346862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:08:50.682485+0000","scrub_duration":0.036164746999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.037996+0000","last_change":"2023-08-22T23:44:58.037996+0000","last_active":"2023-08-22T23:44:58.037996+0000","last_peered":"2023-08-22T23:44:58.037996+0000","last_clean":"2023-08-22T23:44:58.037996+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:58.037996+0000","last_undegraded":"2023-08-22T23:44:58.037996+0000","last_fullsized":"2023-08-22T23:44:58.037996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.035322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.035322+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.035322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:02:25.891429+0000","scrub_duration":0.057636233000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.744778+0000","last_change":"2023-08-22T23:44:52.744778+0000","last_active":"2023-08-22T23:44:52.744778+0000","last_peered":"2023-08-22T23:44:52.744778+0000","last_clean":"2023-08-22T23:44:52.744778+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:52.744778+0000","last_undegraded":"2023-08-22T23:44:52.744778+0000","last_fullsized":"2023-08-22T23:44:52.744778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.738185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.738185+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.738185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:02:04.429354+0000","scrub_duration":0.031314258999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:01.868432+0000","last_change":"2023-08-22T23:45:01.868432+0000","last_active":"2023-08-22T23:45:01.868432+0000","last_peered":"2023-08-22T23:45:01.868432+0000","last_clean":"2023-08-22T23:45:01.868432+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:45:01.868432+0000","last_undegraded":"2023-08-22T23:45:01.868432+0000","last_fullsized":"2023-08-22T23:45:01.868432+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:01.865727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:01.865727+0000","last_clean_scrub_stamp":"2023-08-22T23:45:01.865727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:56:14.967946+0000","scrub_duration":0.082296422999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.674168+0000","last_change":"2023-08-22T23:44:53.674168+0000","last_active":"2023-08-22T23:44:53.674168+0000","last_peered":"2023-08-22T23:44:53.674168+0000","last_clean":"2023-08-22T23:44:53.674168+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:53.674168+0000","last_undegraded":"2023-08-22T23:44:53.674168+0000","last_fullsized":"2023-08-22T23:44:53.674168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.668393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.668393+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.668393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:07:44.060301+0000","scrub_duration":0.039868937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.542439+0000","last_change":"2023-08-22T23:44:48.542439+0000","last_active":"2023-08-22T23:44:48.542439+0000","last_peered":"2023-08-22T23:44:48.542439+0000","last_clean":"2023-08-22T23:44:48.542439+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:48.542439+0000","last_undegraded":"2023-08-22T23:44:48.542439+0000","last_fullsized":"2023-08-22T23:44:48.542439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.539506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:42:32.540349+0000","scrub_duration":0.055977747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:20.456307+0000","last_change":"2023-08-22T23:45:20.457468+0000","last_active":"2023-08-22T23:45:20.456307+0000","last_peered":"2023-08-22T23:45:20.456307+0000","last_clean":"2023-08-22T23:45:20.456307+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:45:20.456307+0000","last_undegraded":"2023-08-22T23:45:20.456307+0000","last_fullsized":"2023-08-22T23:45:20.456307+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:20.454769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:20.454769+0000","last_clean_scrub_stamp":"2023-08-22T23:45:20.454769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:03:05.855957+0000","scrub_duration":0.033051955000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:09.338967+0000","last_change":"2023-08-22T23:45:09.340155+0000","last_active":"2023-08-22T23:45:09.338967+0000","last_peered":"2023-08-22T23:45:09.338967+0000","last_clean":"2023-08-22T23:45:09.338967+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:45:09.338967+0000","last_undegraded":"2023-08-22T23:45:09.338967+0000","last_fullsized":"2023-08-22T23:45:09.338967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:09.337450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:09.337450+0000","last_clean_scrub_stamp":"2023-08-22T23:45:09.337450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:26:04.335737+0000","scrub_duration":0.031976879,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.037886+0000","last_change":"2023-08-22T23:44:52.037886+0000","last_active":"2023-08-22T23:44:52.037886+0000","last_peered":"2023-08-22T23:44:52.037886+0000","last_clean":"2023-08-22T23:44:52.037886+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:52.037886+0000","last_undegraded":"2023-08-22T23:44:52.037886+0000","last_fullsized":"2023-08-22T23:44:52.037886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.032042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.032042+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.032042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:04:20.437680+0000","scrub_duration":0.055160028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":889,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.903442+0000","last_change":"2023-08-22T23:44:48.907344+0000","last_active":"2023-08-22T23:44:48.903442+0000","last_peered":"2023-08-22T23:44:48.903442+0000","last_clean":"2023-08-22T23:44:48.903442+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:48.903442+0000","last_undegraded":"2023-08-22T23:44:48.903442+0000","last_fullsized":"2023-08-22T23:44:48.903442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.901951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.901951+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.901951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:56:14.326385+0000","scrub_duration":0.037514621999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.035238+0000","last_change":"2023-08-22T23:44:53.035238+0000","last_active":"2023-08-22T23:44:53.035238+0000","last_peered":"2023-08-22T23:44:53.035238+0000","last_clean":"2023-08-22T23:44:53.035238+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:53.035238+0000","last_undegraded":"2023-08-22T23:44:53.035238+0000","last_fullsized":"2023-08-22T23:44:53.035238+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.032673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.032673+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.032673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:32:17.339073+0000","scrub_duration":0.075927522999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:04.306062+0000","last_change":"2023-08-22T23:45:04.307187+0000","last_active":"2023-08-22T23:45:04.306062+0000","last_peered":"2023-08-22T23:45:04.306062+0000","last_clean":"2023-08-22T23:45:04.306062+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:45:04.306062+0000","last_undegraded":"2023-08-22T23:45:04.306062+0000","last_fullsized":"2023-08-22T23:45:04.306062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:04.304664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:04.304664+0000","last_clean_scrub_stamp":"2023-08-22T23:45:04.304664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:28:11.096423+0000","scrub_duration":0.036375984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":870,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.913093+0000","last_change":"2023-08-22T23:44:58.917636+0000","last_active":"2023-08-22T23:44:58.913093+0000","last_peered":"2023-08-22T23:44:58.913093+0000","last_clean":"2023-08-22T23:44:58.913093+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:58.913093+0000","last_undegraded":"2023-08-22T23:44:58.913093+0000","last_fullsized":"2023-08-22T23:44:58.913093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.911627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.911627+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.911627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:12:11.522411+0000","scrub_duration":0.032531514999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:50.737739+0000","last_change":"2023-08-22T23:44:50.739077+0000","last_active":"2023-08-22T23:44:50.737739+0000","last_peered":"2023-08-22T23:44:50.737739+0000","last_clean":"2023-08-22T23:44:50.737739+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:50.737739+0000","last_undegraded":"2023-08-22T23:44:50.737739+0000","last_fullsized":"2023-08-22T23:44:50.737739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:50.736264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:50.736264+0000","last_clean_scrub_stamp":"2023-08-22T23:44:50.736264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:58:36.038766+0000","scrub_duration":0.044869521000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:47.828710+0000","last_change":"2023-08-22T23:44:47.829841+0000","last_active":"2023-08-22T23:44:47.828710+0000","last_peered":"2023-08-22T23:44:47.828710+0000","last_clean":"2023-08-22T23:44:47.828710+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:47.828710+0000","last_undegraded":"2023-08-22T23:44:47.828710+0000","last_fullsized":"2023-08-22T23:44:47.828710+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:47.827283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:47.827283+0000","last_clean_scrub_stamp":"2023-08-22T23:44:47.827283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:01:52.954243+0000","scrub_duration":0.069352237999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.638075+0000","last_change":"2023-08-22T23:44:51.639230+0000","last_active":"2023-08-22T23:44:51.638075+0000","last_peered":"2023-08-22T23:44:51.638075+0000","last_clean":"2023-08-22T23:44:51.638075+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:51.638075+0000","last_undegraded":"2023-08-22T23:44:51.638075+0000","last_fullsized":"2023-08-22T23:44:51.638075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:51.636674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:51.636674+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.636674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:06:31.966373+0000","scrub_duration":0.052995683000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.072250+0000","last_change":"2023-08-22T23:44:51.072250+0000","last_active":"2023-08-22T23:44:51.072250+0000","last_peered":"2023-08-22T23:44:51.072250+0000","last_clean":"2023-08-22T23:44:51.072250+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:51.072250+0000","last_undegraded":"2023-08-22T23:44:51.072250+0000","last_fullsized":"2023-08-22T23:44:51.072250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:51.066863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:51.066863+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.066863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:51:38.875911+0000","scrub_duration":0.038690479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":878,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.794477+0000","last_change":"2023-08-22T23:44:54.795936+0000","last_active":"2023-08-22T23:44:54.794477+0000","last_peered":"2023-08-22T23:44:54.794477+0000","last_clean":"2023-08-22T23:44:54.794477+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:54.794477+0000","last_undegraded":"2023-08-22T23:44:54.794477+0000","last_fullsized":"2023-08-22T23:44:54.794477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.792995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.792995+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.792995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:03:09.502831+0000","scrub_duration":0.033049222000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":890,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.481421+0000","last_change":"2023-08-22T23:44:52.481421+0000","last_active":"2023-08-22T23:44:52.481421+0000","last_peered":"2023-08-22T23:44:52.481421+0000","last_clean":"2023-08-22T23:44:52.481421+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:52.481421+0000","last_undegraded":"2023-08-22T23:44:52.481421+0000","last_fullsized":"2023-08-22T23:44:52.481421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.478869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.478869+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.478869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:23:01.134961+0000","scrub_duration":0.031767374000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"918'8","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.900114+0000","last_change":"2023-08-22T23:44:52.901399+0000","last_active":"2023-08-22T23:44:52.900114+0000","last_peered":"2023-08-22T23:44:52.900114+0000","last_clean":"2023-08-22T23:44:52.900114+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:52.900114+0000","last_undegraded":"2023-08-22T23:44:52.900114+0000","last_fullsized":"2023-08-22T23:44:52.900114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8","last_scrub_stamp":"2023-08-22T23:44:52.898702+0000","last_deep_scrub":"918'8","last_deep_scrub_stamp":"2023-08-22T23:44:52.898702+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.898702+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:50:16.475293+0000","scrub_duration":0.076325736000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:50.238325+0000","last_change":"2023-08-22T23:44:50.250441+0000","last_active":"2023-08-22T23:44:50.238325+0000","last_peered":"2023-08-22T23:44:50.238325+0000","last_clean":"2023-08-22T23:44:50.238325+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:50.238325+0000","last_undegraded":"2023-08-22T23:44:50.238325+0000","last_fullsized":"2023-08-22T23:44:50.238325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:50.229791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:50.229791+0000","last_clean_scrub_stamp":"2023-08-22T23:44:50.229791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:58:24.193218+0000","scrub_duration":0.21635689899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.008748+0000","last_change":"2023-08-22T23:44:54.009903+0000","last_active":"2023-08-22T23:44:54.008748+0000","last_peered":"2023-08-22T23:44:54.008748+0000","last_clean":"2023-08-22T23:44:54.008748+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:54.008748+0000","last_undegraded":"2023-08-22T23:44:54.008748+0000","last_fullsized":"2023-08-22T23:44:54.008748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.007311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.007311+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.007311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:49:21.630484+0000","scrub_duration":0.043985337999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:59.284695+0000","last_change":"2023-08-22T23:44:59.285871+0000","last_active":"2023-08-22T23:44:59.284695+0000","last_peered":"2023-08-22T23:44:59.284695+0000","last_clean":"2023-08-22T23:44:59.284695+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:59.284695+0000","last_undegraded":"2023-08-22T23:44:59.284695+0000","last_fullsized":"2023-08-22T23:44:59.284695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:59.283191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:59.283191+0000","last_clean_scrub_stamp":"2023-08-22T23:44:59.283191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:16:55.698349+0000","scrub_duration":0.032512915000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"918'78","reported_seq":995,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.503440+0000","last_change":"2023-08-22T23:44:51.507885+0000","last_active":"2023-08-22T23:44:51.503440+0000","last_peered":"2023-08-22T23:44:51.503440+0000","last_clean":"2023-08-22T23:44:51.503440+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:51.503440+0000","last_undegraded":"2023-08-22T23:44:51.503440+0000","last_fullsized":"2023-08-22T23:44:51.503440+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'78","last_scrub_stamp":"2023-08-22T23:44:51.501949+0000","last_deep_scrub":"918'78","last_deep_scrub_stamp":"2023-08-22T23:44:51.501949+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.501949+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:17:02.406666+0000","scrub_duration":0.125489981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:00.062115+0000","last_change":"2023-08-22T23:45:00.062115+0000","last_active":"2023-08-22T23:45:00.062115+0000","last_peered":"2023-08-22T23:45:00.062115+0000","last_clean":"2023-08-22T23:45:00.062115+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:45:00.062115+0000","last_undegraded":"2023-08-22T23:45:00.062115+0000","last_fullsized":"2023-08-22T23:45:00.062115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:00.059445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:00.059445+0000","last_clean_scrub_stamp":"2023-08-22T23:45:00.059445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:10:12.317174+0000","scrub_duration":0.060663304000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:11.450240+0000","last_change":"2023-08-22T23:45:11.450240+0000","last_active":"2023-08-22T23:45:11.450240+0000","last_peered":"2023-08-22T23:45:11.450240+0000","last_clean":"2023-08-22T23:45:11.450240+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:45:11.450240+0000","last_undegraded":"2023-08-22T23:45:11.450240+0000","last_fullsized":"2023-08-22T23:45:11.450240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:11.444179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:11.444179+0000","last_clean_scrub_stamp":"2023-08-22T23:45:11.444179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:06:04.051365+0000","scrub_duration":0.045936974999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:17.402867+0000","last_change":"2023-08-22T23:45:17.402867+0000","last_active":"2023-08-22T23:45:17.402867+0000","last_peered":"2023-08-22T23:45:17.402867+0000","last_clean":"2023-08-22T23:45:17.402867+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:45:17.402867+0000","last_undegraded":"2023-08-22T23:45:17.402867+0000","last_fullsized":"2023-08-22T23:45:17.402867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:17.400248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:17.400248+0000","last_clean_scrub_stamp":"2023-08-22T23:45:17.400248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:47:16.279822+0000","scrub_duration":0.042204931000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.573413+0000","last_change":"2023-08-22T23:44:54.574547+0000","last_active":"2023-08-22T23:44:54.573413+0000","last_peered":"2023-08-22T23:44:54.573413+0000","last_clean":"2023-08-22T23:44:54.573413+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:54.573413+0000","last_undegraded":"2023-08-22T23:44:54.573413+0000","last_fullsized":"2023-08-22T23:44:54.573413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.571987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.571987+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.571987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:24:26.605250+0000","scrub_duration":0.091127818999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:46.514602+0000","last_change":"2023-08-22T23:44:46.515850+0000","last_active":"2023-08-22T23:44:46.514602+0000","last_peered":"2023-08-22T23:44:46.514602+0000","last_clean":"2023-08-22T23:44:46.514602+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:46.514602+0000","last_undegraded":"2023-08-22T23:44:46.514602+0000","last_fullsized":"2023-08-22T23:44:46.514602+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_clean_scrub_stamp":"2023-08-22T23:44:46.513081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:49:45.949593+0000","scrub_duration":0.047647529000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:26.570329+0000","last_change":"2023-08-22T23:45:26.571547+0000","last_active":"2023-08-22T23:45:26.570329+0000","last_peered":"2023-08-22T23:45:26.570329+0000","last_clean":"2023-08-22T23:45:26.570329+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:45:26.570329+0000","last_undegraded":"2023-08-22T23:45:26.570329+0000","last_fullsized":"2023-08-22T23:45:26.570329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:26.568781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:26.568781+0000","last_clean_scrub_stamp":"2023-08-22T23:45:26.568781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:58:57.463721+0000","scrub_duration":0.051628172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.673754+0000","last_change":"2023-08-22T23:44:52.674889+0000","last_active":"2023-08-22T23:44:52.673754+0000","last_peered":"2023-08-22T23:44:52.673754+0000","last_clean":"2023-08-22T23:44:52.673754+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:52.673754+0000","last_undegraded":"2023-08-22T23:44:52.673754+0000","last_fullsized":"2023-08-22T23:44:52.673754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.672298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.672298+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.672298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:59:07.685804+0000","scrub_duration":0.032147282999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:49.599748+0000","last_change":"2023-08-22T23:44:49.600902+0000","last_active":"2023-08-22T23:44:49.599748+0000","last_peered":"2023-08-22T23:44:49.599748+0000","last_clean":"2023-08-22T23:44:49.599748+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:49.599748+0000","last_undegraded":"2023-08-22T23:44:49.599748+0000","last_fullsized":"2023-08-22T23:44:49.599748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:49.598301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:49.598301+0000","last_clean_scrub_stamp":"2023-08-22T23:44:49.598301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:13:36.863120+0000","scrub_duration":0.084724123999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.377506+0000","last_change":"2023-08-22T23:44:53.378666+0000","last_active":"2023-08-22T23:44:53.377506+0000","last_peered":"2023-08-22T23:44:53.377506+0000","last_clean":"2023-08-22T23:44:53.377506+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:53.377506+0000","last_undegraded":"2023-08-22T23:44:53.377506+0000","last_fullsized":"2023-08-22T23:44:53.377506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.376022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.376022+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.376022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:07:11.775246+0000","scrub_duration":0.036018521999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:16.430482+0000","last_change":"2023-08-22T23:45:16.435059+0000","last_active":"2023-08-22T23:45:16.430482+0000","last_peered":"2023-08-22T23:45:16.430482+0000","last_clean":"2023-08-22T23:45:16.430482+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:45:16.430482+0000","last_undegraded":"2023-08-22T23:45:16.430482+0000","last_fullsized":"2023-08-22T23:45:16.430482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:16.428969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:16.428969+0000","last_clean_scrub_stamp":"2023-08-22T23:45:16.428969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:15:17.601180+0000","scrub_duration":0.054305531999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":869,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.896115+0000","last_change":"2023-08-22T23:44:54.897241+0000","last_active":"2023-08-22T23:44:54.896115+0000","last_peered":"2023-08-22T23:44:54.896115+0000","last_clean":"2023-08-22T23:44:54.896115+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:54.896115+0000","last_undegraded":"2023-08-22T23:44:54.896115+0000","last_fullsized":"2023-08-22T23:44:54.896115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.892207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.892207+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.892207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:22:01.531638+0000","scrub_duration":0.03351606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:18.444771+0000","last_change":"2023-08-22T23:45:18.445952+0000","last_active":"2023-08-22T23:45:18.444771+0000","last_peered":"2023-08-22T23:45:18.444771+0000","last_clean":"2023-08-22T23:45:18.444771+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:45:18.444771+0000","last_undegraded":"2023-08-22T23:45:18.444771+0000","last_fullsized":"2023-08-22T23:45:18.444771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:18.443189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:18.443189+0000","last_clean_scrub_stamp":"2023-08-22T23:45:18.443189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:21:33.407169+0000","scrub_duration":0.075346278000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477015,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65888,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305952,"statfs":{"total":96636764160,"available":96569294848,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":19.638999999999999,"5min":16.050999999999998,"15min":18.748999999999999},"min":{"1min":7.3460000000000001,"5min":4.734,"15min":4.734},"max":{"1min":120.896,"5min":120.896,"15min":254.63},"last":10.802},{"interface":"front","average":{"1min":21.446999999999999,"5min":19.369,"15min":21.602},"min":{"1min":6.3570000000000002,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":127.82299999999999,"5min":127.82299999999999,"15min":330.00099999999998},"last":11.951000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":18.699000000000002,"5min":19.863,"15min":20.460000000000001},"min":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":6.1749999999999998},"max":{"1min":96.775000000000006,"5min":175.85499999999999,"15min":176.803},"last":19.652999999999999},{"interface":"front","average":{"1min":22.876000000000001,"5min":24.242000000000001,"15min":23.904},"min":{"1min":7.2439999999999998,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":126.717,"5min":267.399,"15min":277.048},"last":15.519}]},{"osd":3,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":18.198,"5min":16.169,"15min":18.484999999999999},"min":{"1min":6.0919999999999996,"5min":6.0919999999999996,"15min":5.766},"max":{"1min":106.98699999999999,"5min":112.148,"15min":332.09199999999998},"last":22.486000000000001},{"interface":"front","average":{"1min":26.771999999999998,"5min":21.484999999999999,"15min":24.721},"min":{"1min":9.8439999999999994,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":221.70500000000001,"5min":221.70500000000001,"15min":221.70500000000001},"last":13.941000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":17.216000000000001,"5min":19.616,"15min":23.817},"min":{"1min":9.3539999999999992,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":71.573999999999998,"5min":107.697,"15min":333.517},"last":16.582999999999998},{"interface":"front","average":{"1min":18.596,"5min":23.498000000000001,"15min":27.713000000000001},"min":{"1min":9.5150000000000006,"5min":9.5150000000000006,"15min":8.5779999999999994},"max":{"1min":67.555999999999997,"5min":126.54000000000001,"15min":342.31999999999999},"last":14.861000000000001}]},{"osd":5,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":23.25,"5min":21.058,"15min":22.721},"min":{"1min":9.6099999999999994,"5min":7.1609999999999996,"15min":6.0869999999999997},"max":{"1min":81.114000000000004,"5min":104.696,"15min":326.71300000000002},"last":18.584},{"interface":"front","average":{"1min":20.375,"5min":19.539000000000001,"15min":25.965},"min":{"1min":7.1680000000000001,"5min":7.1680000000000001,"15min":7.1680000000000001},"max":{"1min":85.491,"5min":99.808000000000007,"15min":250.43600000000001},"last":12.917999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":22.710999999999999,"5min":20.698,"15min":25.279},"min":{"1min":11.891999999999999,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":82.162000000000006,"5min":82.162000000000006,"15min":267.84800000000001},"last":23.524999999999999},{"interface":"front","average":{"1min":27.213999999999999,"5min":26.039000000000001,"15min":28.754000000000001},"min":{"1min":12.481,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":104.075,"5min":105.946,"15min":255.58699999999999},"last":20.399000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":25.402000000000001,"5min":27.661999999999999,"15min":31.119},"min":{"1min":12.215999999999999,"5min":11.016,"15min":10.459},"max":{"1min":76.296999999999997,"5min":146.363,"15min":274.60899999999998},"last":21.081},{"interface":"front","average":{"1min":18.678999999999998,"5min":20.068000000000001,"15min":22.187999999999999},"min":{"1min":8.4269999999999996,"5min":8.4269999999999996,"15min":7.8300000000000001},"max":{"1min":84.183999999999997,"5min":114.447,"15min":194.761},"last":17.515000000000001}]}]},{"osd":3,"up_from":16,"seq":68719477016,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83544,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94288296,"statfs":{"total":96636764160,"available":96551215104,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":15.458,"5min":16.449999999999999,"15min":18.245999999999999},"min":{"1min":6.625,"5min":5.6189999999999998,"15min":5.0359999999999996},"max":{"1min":60.25,"5min":153.98599999999999,"15min":220.95400000000001},"last":9.1760000000000002},{"interface":"front","average":{"1min":20.081,"5min":23.474,"15min":25.812000000000001},"min":{"1min":10.656000000000001,"5min":9.2579999999999991,"15min":6.3620000000000001},"max":{"1min":65.772000000000006,"5min":215.34800000000001,"15min":356.75599999999997},"last":12.897}]},{"osd":1,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":13.866,"5min":16.361000000000001,"15min":19.143000000000001},"min":{"1min":6.2240000000000002,"5min":5.218,"15min":5.141},"max":{"1min":59.353999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":8.1300000000000008},{"interface":"front","average":{"1min":14.013,"5min":16.992000000000001,"15min":19.372},"min":{"1min":8.0760000000000005,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":63.573,"5min":190.49799999999999,"15min":374.01799999999997},"last":11.196}]},{"osd":2,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":17.425000000000001,"5min":22.100000000000001,"15min":21.399000000000001},"min":{"1min":8.0069999999999997,"5min":6.524,"15min":6.524},"max":{"1min":64.561000000000007,"5min":198.649,"15min":233.745},"last":7.3310000000000004},{"interface":"front","average":{"1min":17.495999999999999,"5min":20.888000000000002,"15min":22.417999999999999},"min":{"1min":7.8179999999999996,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":70.597999999999999,"5min":193.53999999999999,"15min":241.23500000000001},"last":18.715}]},{"osd":4,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":17.477,"5min":21.974,"15min":27.140000000000001},"min":{"1min":8.7720000000000002,"5min":8.7720000000000002,"15min":6.6230000000000002},"max":{"1min":72.275000000000006,"5min":199.42400000000001,"15min":370.84399999999999},"last":10.566000000000001},{"interface":"front","average":{"1min":19.637,"5min":24.417000000000002,"15min":30.408999999999999},"min":{"1min":8.4450000000000003,"5min":6.726,"15min":6.726},"max":{"1min":89.596999999999994,"5min":231.53,"15min":372.45299999999997},"last":14.506}]},{"osd":5,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":21.088999999999999,"15min":24.824999999999999},"min":{"1min":9.6869999999999994,"5min":9.6869999999999994,"15min":8.5419999999999998},"max":{"1min":67.260999999999996,"5min":175.429,"15min":241.86099999999999},"last":13.693},{"interface":"front","average":{"1min":18.904,"5min":20.405000000000001,"15min":24.536000000000001},"min":{"1min":8.6530000000000005,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":74.754999999999995,"5min":212.09,"15min":289.34300000000002},"last":17.861000000000001}]},{"osd":6,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":20.495000000000001,"5min":22.859999999999999,"15min":27.015999999999998},"min":{"1min":11.388999999999999,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":68.186000000000007,"5min":213.15000000000001,"15min":246.30000000000001},"last":12.263999999999999},{"interface":"front","average":{"1min":25.173999999999999,"5min":28.638000000000002,"15min":27.481999999999999},"min":{"1min":12.978,"5min":8.1669999999999998,"15min":7.7830000000000004},"max":{"1min":73.796000000000006,"5min":215.41300000000001,"15min":215.41300000000001},"last":15.138}]},{"osd":7,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":22.321999999999999,"5min":22.753,"15min":28.486000000000001},"min":{"1min":12.140000000000001,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":76.263999999999996,"5min":206.708,"15min":417.863},"last":19.556000000000001},{"interface":"front","average":{"1min":22.802,"5min":27.838999999999999,"15min":28.358000000000001},"min":{"1min":13.613,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":71.433999999999997,"5min":229.85900000000001,"15min":229.85900000000001},"last":16.882000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477015,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65500,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94306340,"statfs":{"total":96636764160,"available":96569692160,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":10.722,"5min":15.116,"15min":16.167999999999999},"min":{"1min":6.5270000000000001,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":25.141999999999999,"5min":204.17500000000001,"15min":208.114},"last":11.032999999999999},{"interface":"front","average":{"1min":15.356999999999999,"5min":20.913,"15min":19.713000000000001},"min":{"1min":9.109,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":31.975000000000001,"5min":205.833,"15min":205.833},"last":15.335000000000001}]},{"osd":1,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":12.912000000000001,"5min":16.635999999999999,"15min":17.698},"min":{"1min":6.9400000000000004,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":57.646000000000001,"5min":142.42599999999999,"15min":218.06200000000001},"last":7.1710000000000003},{"interface":"front","average":{"1min":19.082999999999998,"5min":21.914000000000001,"15min":21.427},"min":{"1min":10.345000000000001,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":62.720999999999997,"5min":145.83000000000001,"15min":230.108},"last":11.829000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:44:54 2023","interfaces":[{"interface":"back","average":{"1min":11.27,"5min":21.934999999999999,"15min":20.477},"min":{"1min":6.9749999999999996,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":23.353999999999999,"5min":292.49000000000001,"15min":292.49000000000001},"last":7.9900000000000002},{"interface":"front","average":{"1min":20.030000000000001,"5min":28.835999999999999,"15min":25.010999999999999},"min":{"1min":10.519,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":65.405000000000001,"5min":304.017,"15min":306.26299999999998},"last":15.813000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":15.063000000000001,"5min":19.024999999999999,"15min":20.800000000000001},"min":{"1min":9.0210000000000008,"5min":8.7170000000000005,"15min":7.3869999999999996},"max":{"1min":33.100999999999999,"5min":157.559,"15min":187.357},"last":8.8320000000000007},{"interface":"front","average":{"1min":15.558,"5min":18.388999999999999,"15min":20.344000000000001},"min":{"1min":8.9309999999999992,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":21.797999999999998,"5min":146.64599999999999,"15min":206.59100000000001},"last":16.562999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":18.018000000000001,"5min":22.248000000000001,"15min":23.747},"min":{"1min":11.853,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":33.109000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":9.4740000000000002},{"interface":"front","average":{"1min":16.687000000000001,"5min":21.004000000000001,"15min":22.106000000000002},"min":{"1min":8.234,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":30.888000000000002,"5min":181.24199999999999,"15min":181.24199999999999},"last":18.132999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":17.91,"5min":24.329000000000001,"15min":23.096},"min":{"1min":11.965999999999999,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":30.242000000000001,"5min":189.55000000000001,"15min":189.55000000000001},"last":12.525},{"interface":"front","average":{"1min":17.471,"5min":21.184000000000001,"15min":22.276},"min":{"1min":9.7170000000000005,"5min":9.2210000000000001,"15min":8.8819999999999997},"max":{"1min":29.152999999999999,"5min":129.928,"15min":130.45599999999999},"last":20.033999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":23.861999999999998,"5min":26.873000000000001,"15min":28.923999999999999},"min":{"1min":11.734,"5min":11.529999999999999,"15min":9.9939999999999998},"max":{"1min":67.033000000000001,"5min":195.655,"15min":402.06099999999998},"last":14.803000000000001},{"interface":"front","average":{"1min":16.623999999999999,"5min":21.030000000000001,"15min":27.128},"min":{"1min":7.8840000000000003,"5min":7.8840000000000003,"15min":7.8840000000000003},"max":{"1min":34.084000000000003,"5min":158.26499999999999,"15min":427.274},"last":17.347999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477016,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66152,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305688,"statfs":{"total":96636764160,"available":96569024512,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":11.734,"5min":13.788,"15min":15.262},"min":{"1min":7.1319999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":30.100999999999999,"5min":59.094999999999999,"15min":164.69999999999999},"last":17.387},{"interface":"front","average":{"1min":12.417,"5min":17.076000000000001,"15min":17.960999999999999},"min":{"1min":7.431,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":28.806000000000001,"5min":150.68199999999999,"15min":256.70100000000002},"last":11.574}]},{"osd":2,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":18.806000000000001,"15min":20.181999999999999},"min":{"1min":9.2899999999999991,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":19.698,"5min":133.678,"15min":255.041},"last":14.545999999999999},{"interface":"front","average":{"1min":12.794,"5min":18.503,"15min":20.234000000000002},"min":{"1min":8.4459999999999997,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":23.007999999999999,"5min":122.78700000000001,"15min":266.08600000000001},"last":13.041}]},{"osd":3,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":10.009,"5min":18.423999999999999,"15min":18.460999999999999},"min":{"1min":6.7750000000000004,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":16.966999999999999,"5min":140.357,"15min":269.73200000000003},"last":7.9820000000000002},{"interface":"front","average":{"1min":16.195,"5min":20.917999999999999,"15min":21.375},"min":{"1min":10.243,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":29.484000000000002,"5min":124.69499999999999,"15min":253.67099999999999},"last":12.305999999999999}]},{"osd":4,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":13.577999999999999,"5min":17.079999999999998,"15min":21.831},"min":{"1min":7.3380000000000001,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":19.238,"5min":101.029,"15min":270.06900000000002},"last":20.431999999999999},{"interface":"front","average":{"1min":12.122,"5min":19.228000000000002,"15min":22.085000000000001},"min":{"1min":6.9649999999999999,"5min":6.9649999999999999,"15min":6.9649999999999999},"max":{"1min":19.108000000000001,"5min":137.608,"15min":254.87899999999999},"last":9.202}]},{"osd":5,"last update":"Tue Aug 22 23:45:04 2023","interfaces":[{"interface":"back","average":{"1min":27.023,"5min":25.725999999999999,"15min":25.164000000000001},"min":{"1min":8.9079999999999995,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":263.81599999999997,"5min":263.81599999999997,"15min":287.80200000000002},"last":10.178000000000001},{"interface":"front","average":{"1min":29.484999999999999,"5min":25.931000000000001,"15min":26.088000000000001},"min":{"1min":10.371,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":272.935,"5min":272.935,"15min":272.935},"last":19.57}]},{"osd":6,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":18.088000000000001,"5min":25.870999999999999,"15min":24.484000000000002},"min":{"1min":13.003,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":28.029,"5min":167.541,"15min":276.935},"last":18.847999999999999},{"interface":"front","average":{"1min":15.387,"5min":20.478000000000002,"15min":23.152000000000001},"min":{"1min":7.8179999999999996,"5min":7.8179999999999996,"15min":7.8179999999999996},"max":{"1min":26.350999999999999,"5min":109.886,"15min":265.17399999999998},"last":15.273999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:45:04 2023","interfaces":[{"interface":"back","average":{"1min":18.571000000000002,"5min":27.088999999999999,"15min":30.423999999999999},"min":{"1min":14.311999999999999,"5min":10.805999999999999,"15min":7.1760000000000002},"max":{"1min":32.917000000000002,"5min":184.333,"15min":770.02200000000005},"last":18.099},{"interface":"front","average":{"1min":15.192,"5min":24.359999999999999,"15min":30.084},"min":{"1min":10.214,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":21.288,"5min":154.00800000000001,"15min":834.71799999999996},"last":13.734999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509721,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65500,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94306340,"statfs":{"total":96636764160,"available":96569692160,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":15.823,"5min":15.853999999999999,"15min":16.640999999999998},"min":{"1min":5.774,"5min":5.774,"15min":4.5570000000000004},"max":{"1min":77.439999999999998,"5min":101.271,"15min":190.38800000000001},"last":17.823},{"interface":"front","average":{"1min":25.606000000000002,"5min":23.318999999999999,"15min":21.681000000000001},"min":{"1min":9.0399999999999991,"5min":9.0399999999999991,"15min":4.9699999999999998},"max":{"1min":120.398,"5min":122.059,"15min":215.38800000000001},"last":18.536999999999999}]},{"osd":1,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":18.706,"5min":15.819000000000001,"15min":16.527999999999999},"min":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.1100000000000003},"max":{"1min":46.259999999999998,"5min":77.727999999999994,"15min":222.346},"last":9.7509999999999994},{"interface":"front","average":{"1min":21.076000000000001,"5min":18.771999999999998,"15min":19.385000000000002},"min":{"1min":7.4039999999999999,"5min":7.4039999999999999,"15min":6.0519999999999996},"max":{"1min":53.518000000000001,"5min":137.86799999999999,"15min":216.02000000000001},"last":19.248999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":17.227,"5min":16.443999999999999,"15min":18.135000000000002},"min":{"1min":7.0960000000000001,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":53.883000000000003,"5min":171.38399999999999,"15min":259.274},"last":13.756},{"interface":"front","average":{"1min":26.795000000000002,"5min":20.93,"15min":21.888000000000002},"min":{"1min":9.859,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":137.751,"5min":170.02600000000001,"15min":223.89699999999999},"last":14.452}]},{"osd":3,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":21.032,"5min":21.359999999999999,"15min":24.524999999999999},"min":{"1min":9.3279999999999994,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":51.694000000000003,"5min":208.94800000000001,"15min":352.79500000000002},"last":15.162000000000001},{"interface":"front","average":{"1min":20.038,"5min":22.895,"15min":22.268000000000001},"min":{"1min":10.686,"5min":9.7870000000000008,"15min":7.4370000000000003},"max":{"1min":46.271999999999998,"5min":178.21600000000001,"15min":193.483},"last":19.983000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":16.306999999999999,"5min":16.852,"15min":19.088999999999999},"min":{"1min":8.1959999999999997,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":40.009999999999998,"5min":91.307000000000002,"15min":151.41399999999999},"last":8.4390000000000001},{"interface":"front","average":{"1min":22.344000000000001,"5min":22.126000000000001,"15min":26.108000000000001},"min":{"1min":10.922000000000001,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":50.005000000000003,"5min":92.557000000000002,"15min":156.40100000000001},"last":12.196}]},{"osd":5,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":23.013999999999999,"5min":22.649999999999999,"15min":25.038},"min":{"1min":11.438000000000001,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":77.650000000000006,"5min":183.268,"15min":249.37299999999999},"last":20.692},{"interface":"front","average":{"1min":21.149999999999999,"5min":21.687000000000001,"15min":23.161999999999999},"min":{"1min":8.9149999999999991,"5min":8.9149999999999991,"15min":7.3849999999999998},"max":{"1min":62.554000000000002,"5min":146.96199999999999,"15min":215.76499999999999},"last":11.016}]},{"osd":7,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":26.050999999999998,"5min":22.292000000000002,"15min":25.626999999999999},"min":{"1min":10.052,"5min":10.052,"15min":9.3559999999999999},"max":{"1min":82.512,"5min":91.301000000000002,"15min":457.08199999999999},"last":12.962999999999999},{"interface":"front","average":{"1min":27.477,"5min":23.448,"15min":26.001999999999999},"min":{"1min":13.436999999999999,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":81.609999999999999,"5min":155.268,"15min":353.75900000000001},"last":21.495000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509719,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88296,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283544,"statfs":{"total":96636764160,"available":96546349056,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":9.5980000000000008,"5min":19.257999999999999,"15min":21.283000000000001},"min":{"1min":5.9900000000000002,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":17.904,"5min":141.732,"15min":235.06100000000001},"last":14.858000000000001},{"interface":"front","average":{"1min":14.381,"5min":22.114999999999998,"15min":25.699000000000002},"min":{"1min":6.8630000000000004,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":35.685000000000002,"5min":189.517,"15min":395.15300000000002},"last":6.5490000000000004}]},{"osd":1,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":15.704000000000001,"5min":24.719000000000001,"15min":23.050000000000001},"min":{"1min":8.1270000000000007,"5min":6.7149999999999999,"15min":6.0039999999999996},"max":{"1min":46.531999999999996,"5min":241.87799999999999,"15min":347.69200000000001},"last":7.8339999999999996},{"interface":"front","average":{"1min":20.838000000000001,"5min":32.209000000000003,"15min":29.765999999999998},"min":{"1min":11.089,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":49.670000000000002,"5min":252.441,"15min":350.66199999999998},"last":12.901999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":15.763,"5min":21.625,"15min":22.677},"min":{"1min":7.0289999999999999,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":47.415999999999997,"5min":219.37899999999999,"15min":234.05699999999999},"last":15.541},{"interface":"front","average":{"1min":16.007999999999999,"5min":22.385000000000002,"15min":28.353000000000002},"min":{"1min":6.5629999999999997,"5min":6.5629999999999997,"15min":6.1210000000000004},"max":{"1min":37.369,"5min":214.50700000000001,"15min":467.80500000000001},"last":11.507}]},{"osd":3,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":16.234000000000002,"5min":22.823,"15min":26.353000000000002},"min":{"1min":8.1280000000000001,"5min":7.0449999999999999,"15min":6.899},"max":{"1min":55.274999999999999,"5min":161.05199999999999,"15min":284.47899999999998},"last":17.622},{"interface":"front","average":{"1min":18.247,"5min":28.542999999999999,"15min":29.581},"min":{"1min":8.3350000000000009,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":40.488999999999997,"5min":226.25700000000001,"15min":339.51499999999999},"last":14.532}]},{"osd":5,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":28.321000000000002,"5min":31.291,"15min":30.677},"min":{"1min":9.5660000000000007,"5min":8.5909999999999993,"15min":7.2279999999999998},"max":{"1min":155.17599999999999,"5min":238.57300000000001,"15min":380.863},"last":16.251000000000001},{"interface":"front","average":{"1min":27.881,"5min":29.675000000000001,"15min":33.774000000000001},"min":{"1min":10.247999999999999,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":158.03899999999999,"5min":215.928,"15min":342.858},"last":19.317}]},{"osd":6,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":23.414000000000001,"5min":29.004999999999999,"15min":30.018999999999998},"min":{"1min":12.186999999999999,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":70.799000000000007,"5min":217,"15min":249.29599999999999},"last":10.074999999999999},{"interface":"front","average":{"1min":19.931000000000001,"5min":23.629000000000001,"15min":28.702999999999999},"min":{"1min":7.6280000000000001,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":74.850999999999999,"5min":220.87100000000001,"15min":468.887},"last":16.867000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:45:17 2023","interfaces":[{"interface":"back","average":{"1min":18.567,"5min":22.748999999999999,"15min":28.390000000000001},"min":{"1min":9.4450000000000003,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":52.113999999999997,"5min":187.30099999999999,"15min":333.20699999999999},"last":10.875},{"interface":"front","average":{"1min":22.135000000000002,"5min":26.247,"15min":29.818000000000001},"min":{"1min":12.84,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":56.811,"5min":197.05600000000001,"15min":368.08699999999999},"last":12.121}]}]},{"osd":7,"up_from":14,"seq":60129542426,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87640,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94284200,"statfs":{"total":96636764160,"available":96547020800,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":11.638,"5min":16.164999999999999,"15min":19.459},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":4.3040000000000003},"max":{"1min":19.277999999999999,"5min":307.584,"15min":307.584},"last":10.198},{"interface":"front","average":{"1min":15.122,"5min":17.684999999999999,"15min":20.268999999999998},"min":{"1min":8.9990000000000006,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":23.640999999999998,"5min":261.58999999999997,"15min":317.81},"last":16.239999999999998}]},{"osd":1,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":12.972,"5min":14.698,"15min":23.751999999999999},"min":{"1min":6.819,"5min":5.077,"15min":5.077},"max":{"1min":22.248000000000001,"5min":79.575000000000003,"15min":319.34500000000003},"last":12.226000000000001},{"interface":"front","average":{"1min":11.390000000000001,"5min":12.893000000000001,"15min":19.420000000000002},"min":{"1min":6.96,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":17.920000000000002,"5min":78.543999999999997,"15min":195.78999999999999},"last":9.6379999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":16.155000000000001,"5min":16.762,"15min":20.832999999999998},"min":{"1min":8.6720000000000006,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":21.527000000000001,"5min":148.97800000000001,"15min":304.25},"last":14.468},{"interface":"front","average":{"1min":13.529999999999999,"5min":17.765000000000001,"15min":21.911000000000001},"min":{"1min":7.4160000000000004,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":21.960999999999999,"5min":151.46700000000001,"15min":264.49400000000003},"last":18.385000000000002}]},{"osd":3,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":11.039,"5min":17.289000000000001,"15min":21.934000000000001},"min":{"1min":7.7130000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":16.832999999999998,"5min":154.44200000000001,"15min":243.33699999999999},"last":11.5},{"interface":"front","average":{"1min":16.942,"5min":21.024999999999999,"15min":26.138999999999999},"min":{"1min":7.3049999999999997,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":30.774000000000001,"5min":246.798,"15min":337.09699999999998},"last":19.895}]},{"osd":4,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":19.015000000000001,"15min":27.401},"min":{"1min":7.6020000000000003,"5min":7.4480000000000004,"15min":7.4480000000000004},"max":{"1min":23.989999999999998,"5min":111.015,"15min":265.17899999999997},"last":17.677},{"interface":"front","average":{"1min":15.244999999999999,"5min":17.884,"15min":22.994},"min":{"1min":11.186,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":21.315999999999999,"5min":117.06100000000001,"15min":192.58600000000001},"last":12.968}]},{"osd":5,"last update":"Tue Aug 22 23:45:06 2023","interfaces":[{"interface":"back","average":{"1min":15.492000000000001,"5min":20.783000000000001,"15min":28.716999999999999},"min":{"1min":11.125999999999999,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":23.506,"5min":247.898,"15min":342.53100000000001},"last":13.66},{"interface":"front","average":{"1min":18.222000000000001,"5min":21.045000000000002,"15min":26.207999999999998},"min":{"1min":10.242000000000001,"5min":8.8770000000000007,"15min":8.5809999999999995},"max":{"1min":33.665999999999997,"5min":135.31399999999999,"15min":271.47500000000002},"last":19.079000000000001}]},{"osd":6,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":18.920000000000002,"5min":19.762,"15min":24.591000000000001},"min":{"1min":12.364000000000001,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":24.834,"5min":144.21600000000001,"15min":228.43100000000001},"last":20.73},{"interface":"front","average":{"1min":19.818000000000001,"5min":20.763000000000002,"15min":27.73},"min":{"1min":8.8559999999999999,"5min":8.8559999999999999,"15min":8.8559999999999999},"max":{"1min":66.206000000000003,"5min":99.783000000000001,"15min":262.45800000000003},"last":16.945}]}]},{"osd":5,"up_from":14,"seq":60129542423,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83936,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287904,"statfs":{"total":96636764160,"available":96550813696,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":19.690000000000001,"5min":17.018000000000001,"15min":19.709},"min":{"1min":6.0819999999999999,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":169.48699999999999,"5min":169.48699999999999,"15min":220.017},"last":13.826000000000001},{"interface":"front","average":{"1min":25.696999999999999,"5min":23.117000000000001,"15min":25.722000000000001},"min":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":5.9329999999999998},"max":{"1min":168.82400000000001,"5min":168.82400000000001,"15min":264.11900000000003},"last":16.661999999999999}]},{"osd":1,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":14.484,"5min":14.842000000000001,"15min":17.66},"min":{"1min":6.5830000000000002,"5min":6.5119999999999996,"15min":5.9379999999999997},"max":{"1min":40.689,"5min":102.66200000000001,"15min":257.60199999999998},"last":10.542999999999999},{"interface":"front","average":{"1min":22.949999999999999,"5min":21.742000000000001,"15min":23.655000000000001},"min":{"1min":7.0999999999999996,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":150.47300000000001,"5min":150.47300000000001,"15min":230.73400000000001},"last":17.335000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":20.837,"5min":17.885999999999999,"15min":19.574000000000002},"min":{"1min":7.8120000000000003,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":82.043000000000006,"5min":116.45,"15min":248.74799999999999},"last":15.878},{"interface":"front","average":{"1min":18.931000000000001,"5min":17.198,"15min":22.306999999999999},"min":{"1min":7.9569999999999999,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":82.939999999999998,"5min":99.498000000000005,"15min":256.94400000000002},"last":11.223000000000001}]},{"osd":3,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":22.113,"5min":25,"15min":29.079000000000001},"min":{"1min":9.8559999999999999,"5min":6.9859999999999998,"15min":6.5739999999999998},"max":{"1min":71.793999999999997,"5min":231.99799999999999,"15min":257.23500000000001},"last":12.285},{"interface":"front","average":{"1min":26.582999999999998,"5min":26.600000000000001,"15min":23.663},"min":{"1min":9.6709999999999994,"5min":6.827,"15min":6.7489999999999997},"max":{"1min":132.06299999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":18.032}]},{"osd":4,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":22.515000000000001,"5min":23.562000000000001,"15min":25.693000000000001},"min":{"1min":10.301,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":71.150000000000006,"5min":143.23500000000001,"15min":212.77500000000001},"last":14.532999999999999},{"interface":"front","average":{"1min":21.497,"5min":23.329999999999998,"15min":24.059999999999999},"min":{"1min":10.753,"5min":7.5389999999999997,"15min":7.2599999999999998},"max":{"1min":53.722999999999999,"5min":156.947,"15min":273.58699999999999},"last":8.1370000000000005}]},{"osd":6,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":29.331,"5min":28.152000000000001,"15min":30.099},"min":{"1min":9.0649999999999995,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":149.81399999999999,"5min":228.626,"15min":241.26499999999999},"last":18.696000000000002},{"interface":"front","average":{"1min":28.795000000000002,"5min":25.315000000000001,"15min":25.981999999999999},"min":{"1min":10.907,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":121.235,"5min":247.06800000000001,"15min":247.06800000000001},"last":15.244999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":26.649000000000001,"5min":22.806000000000001,"15min":25.640000000000001},"min":{"1min":11.548999999999999,"5min":8.6859999999999999,"15min":8.2560000000000002},"max":{"1min":97.043999999999997,"5min":121.524,"15min":317.44600000000003},"last":19.469000000000001},{"interface":"front","average":{"1min":24.292000000000002,"5min":25.937999999999999,"15min":26.905999999999999},"min":{"1min":10.48,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":69.792000000000002,"5min":201.81700000000001,"15min":361.93000000000001},"last":11.866}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:45:32.770 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-08-22T23:23:27.428724+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=23, tm_sec=27, tm_wday=1, tm_yday=234, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=22, tm_hour=23, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=234, tm_isdst=0) 2023-08-22T23:45:32.770 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-08-22T23:45:33.623 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:18.907 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.628 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:18.912 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.642 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:18.925 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.671 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:18.955 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.676 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:18.960 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.681 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:18.978 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.682 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:18.980 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.688 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:18.972 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:33.694 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:18.991 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.826 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:23.110 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.829 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:23.113 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.843 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:23.127 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.883 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:23.167 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.884 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:23.168 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.887 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:23.185 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.890 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:23.188 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.899 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:23.183 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:37.903 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:23.201 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.668 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:28.952 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.669 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:28.953 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.680 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:28.964 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.714 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:28.997 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.714 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:29.012 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.717 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:29.001 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.718 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:29.015 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.731 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:29.028 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:43.732 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:29.015 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.730 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:34.013 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.733 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:34.017 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.749 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:34.033 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.782 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:34.066 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.786 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:34.084 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.787 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:34.070 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.792 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:34.089 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.803 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:34.087 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:48.807 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:34.104 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:52.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-22T23:45:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.881+0000 7fd869ccc640 10 monclient: get_monmap_and_config 2023-08-22T23:45:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.881+0000 7fd869ccc640 10 monclient: build_initial_monmap 2023-08-22T23:45:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.881+0000 7fd869ccc640 10 monclient: monmap: 2023-08-22T23:45:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:45:52.882459+0000 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:45:52.882459+0000 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:45:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:45:52.884 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:45:52.884 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.881+0000 7fd869ccc640 1 Processor -- start 2023-08-22T23:45:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.881+0000 7fd869ccc640 1 -- start start 2023-08-22T23:45:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.881+0000 7fd869ccc640 10 monclient: init 2023-08-22T23:45:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-08-22T23:45:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8640a7b90 addr v1:172.21.15.52:6789/0 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8640b1380 addr v1:172.21.15.29:6789/0 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7f70 0x7fd86414ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient(hunting): picked mon.noname-b con 0x7fd8640a7f70 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640a9b00 con 0x7fd8640b1380 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640a9de0 con 0x7fd8640a7b90 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640aa0c0 con 0x7fd8640a7f70 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.882+0000 7fd869ccc640 10 monclient(hunting): authenticate will time out at 2917.133019s 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd8637fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fd8640a7b90 0x7fd86413e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.29:44104/0 (socket says 172.21.15.29:44104) 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd8637fe640 1 -- 172.21.15.29:0/1106360557 learned_addr learned my addr 172.21.15.29:0/1106360557 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd863fff640 1 --2- 172.21.15.29:0/1106360557 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7f70 0x7fd86414ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd863fff640 10 monclient(hunting): get_auth_request con 0x7fd8640a7f70 auth_method 0 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd863fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.883+0000 7fd863fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.884+0000 7fd863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.884+0000 7fd863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.884+0000 7fd863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:52.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121003839 0 0) 0x7fd8640a9de0 con 0x7fd8640a7b90 2023-08-22T23:45:52.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): _negotiate 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): my global_id is 19672 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd84c003520 con 0x7fd8640a7b90 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458182888 0 0) 0x7fd8640a9b00 con 0x7fd8640b1380 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): _negotiate 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 10 monclient(hunting): my global_id is 14850 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.885+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8640a9de0 con 0x7fd8640b1380 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 10 monclient(hunting): handle_auth_done global_id 16751 payload 274 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 1 -- 172.21.15.29:0/1106360557 >> v1:172.21.15.52:6789/0 conn(0x7fd8640a7b90 legacy=0x7fd86413e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 1 -- 172.21.15.29:0/1106360557 >> v1:172.21.15.29:6789/0 conn(0x7fd8640b1380 legacy=0x7fd8641417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 10 monclient: _finish_hunting 0 2023-08-22T23:45:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 1 monclient: found mon.noname-b 2023-08-22T23:45:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.29:3301/0 2023-08-22T23:45:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.886+0000 7fd863fff640 1 -- 172.21.15.29:0/1106360557 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8641412c0 con 0x7fd8640a7f70 2023-08-22T23:45:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.888+0000 7fd863fff640 1 --2- 172.21.15.29:0/1106360557 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7f70 0x7fd86414ca40 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009dc0 tx=0x7fd85c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6a2ead44945b985 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c02ccd0 con 0x7fd8640a7f70 2023-08-22T23:45:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-08-22T23:45:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: dump: 2023-08-22T23:45:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: _finish_auth 0 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: _check_auth_tickets 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c02ce30 con 0x7fd8640a7f70 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 5 monclient: authenticate success, global_id 16751 2023-08-22T23:45:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 1 -- 172.21.15.29:0/1106360557 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c034660 con 0x7fd8640a7f70 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd8627fc640 10 monclient: dump: 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 10 monclient: get_monmap_and_config success 2023-08-22T23:45:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 4 set_mon_vals no callback set 2023-08-22T23:45:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 10 monclient: shutdown 2023-08-22T23:45:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 1 -- 172.21.15.29:0/1106360557 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7f70 msgr2=0x7fd86414ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1106360557 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7f70 0x7fd86414ca40 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009dc0 tx=0x7fd85c02bf20 comp rx=0 tx=0).stop 2023-08-22T23:45:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 1 -- 172.21.15.29:0/1106360557 shutdown_connections 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1106360557 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7f70 0x7fd86414ca40 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 1 -- 172.21.15.29:0/1106360557 >> 172.21.15.29:0/1106360557 conn(0x7fd8640b6a00 msgr2=0x7fd8640b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.892+0000 7fd869ccc640 1 -- 172.21.15.29:0/1106360557 shutdown_connections 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.893+0000 7fd869ccc640 1 -- 172.21.15.29:0/1106360557 wait complete. 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.893+0000 7fd869ccc640 10 monclient: build_initial_monmap 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.893+0000 7fd869ccc640 10 monclient: monmap: 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:45:52.902 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.893+0000 7fd869ccc640 1 librados: starting msgr at 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.893+0000 7fd869ccc640 1 librados: starting objecter 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 Processor -- start 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 -- start start 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 librados: setting wanted keys 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 librados: calling monclient init 2023-08-22T23:45:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient: init 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient: _add_conns ranks=[2,1,0] 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 0x7fd864142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): picked mon.noname-c con 0x7fd8640a7b90 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8640a7f70 0x7fd8641d9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): picked mon.noname-b con 0x7fd8640a7f70 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8640b1380 0x7fd8641d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): picked mon.noname-a con 0x7fd8640b1380 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641d9470 con 0x7fd8640b1380 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641439e0 con 0x7fd8640a7f70 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd864143c50 con 0x7fd8640a7b90 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.894+0000 7fd869ccc640 10 monclient(hunting): authenticate will time out at 2917.145322s 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 0x7fd864142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd863fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8640b1380 0x7fd8641d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 0x7fd864142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3301/0 says I am v2:172.21.15.29:54950/0 (socket says 172.21.15.29:54950) 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 1 -- 172.21.15.29:0/1196979284 learned_addr learned my addr 172.21.15.29:0/1196979284 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 10 monclient(hunting): get_auth_request con 0x7fd8640a7b90 auth_method 0 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd8637fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd863fff640 10 monclient(hunting): get_auth_request con 0x7fd8640b1380 auth_method 0 2023-08-22T23:45:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd863fff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.895+0000 7fd863fff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.896+0000 7fd8637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.896+0000 7fd8637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.896+0000 7fd8637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.896+0000 7fd863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.896+0000 7fd863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.896+0000 7fd863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.897+0000 7fd862ffd640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8640a7f70 0x7fd8641d9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): get_auth_request con 0x7fd8640a7f70 auth_method 0 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.898+0000 7fd862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 10 monclient(hunting): handle_auth_done global_id 16754 payload 824 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 1 -- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8640a7f70 msgr2=0x7fd8641d9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8640a7f70 0x7fd8641d9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 1 -- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8640b1380 msgr2=0x7fd8641d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8640b1380 0x7fd8641d9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 10 monclient: _finish_hunting 0 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 1 monclient: found mon.noname-c 2023-08-22T23:45:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.29:3301/0 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.900+0000 7fd8637fe640 1 -- 172.21.15.29:0/1196979284 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c009a70 con 0x7fd8640a7b90 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.903+0000 7fd8637fe640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 0x7fd864142710 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c530 tx=0x7fd85400c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.903+0000 7fd84bfff640 10 client.?.objecter ms_handle_connect 0x7fd8640a7b90 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.903+0000 7fd84bfff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mon.2 v2:172.21.15.29:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854005ce0 con 0x7fd8640a7b90 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 10 monclient: dump: 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:52.915 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 10 monclient: _finish_auth 0 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 10 monclient: _check_auth_tickets 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd84bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd869ccc640 5 monclient: authenticate success, global_id 16754 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd869ccc640 10 monclient: _renew_subs 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd869ccc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.906+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86413a410 con 0x7fd8640a7b90 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 10 client.16754.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 10 monclient: _renew_subs 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd864144350 con 0x7fd8640a7b90 2023-08-22T23:45:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 1 librados: init done 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mon.2 v2:172.21.15.29:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd854005e40 con 0x7fd8640a7b90 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd84bfff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd861ffb640 4 set_mon_vals no callback set 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mon.2 v2:172.21.15.29:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854007270 con 0x7fd8640a7b90 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd84bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd84bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd84bfff640 10 monclient: dump: 2023-08-22T23:45:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:52.920 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.29:3301/0 2023-08-22T23:45:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.907+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86404dbd0 con 0x7fd8640a7b90 2023-08-22T23:45:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.927+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mon.2 v2:172.21.15.29:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd8540073d0 con 0x7fd8640a7b90 2023-08-22T23:45:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.927+0000 7fd84bfff640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd85006b9e0 0x7fd85006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.928+0000 7fd862ffd640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd85006b9e0 0x7fd85006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.928+0000 7fd862ffd640 10 monclient: get_auth_request con 0x7fd85006b9e0 auth_method 0 2023-08-22T23:45:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.928+0000 7fd862ffd640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd85006b9e0 0x7fd85006dea0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd86413bbb0 tx=0x7fd858009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.928+0000 7fd84bfff640 10 client.16754.objecter ms_handle_connect 0x7fd85006b9e0 2023-08-22T23:45:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.929+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mon.2 v2:172.21.15.29:3301/0 5 ==== osd_map(918..918 src has 827..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fd85405b4e0 con 0x7fd8640a7b90 2023-08-22T23:45:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.929+0000 7fd84bfff640 10 client.16754.objecter ms_dispatch 0x7fd864000b90 osd_map(918..918 src has 827..918) v4 2023-08-22T23:45:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.929+0000 7fd84bfff640 3 client.16754.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:45:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.929+0000 7fd84bfff640 3 client.16754.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:45:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:52.930+0000 7fd84bfff640 20 client.16754.objecter dump_active .. 0 homeless 2023-08-22T23:45:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.180+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mon.2 v2:172.21.15.29:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7fd854053b10 con 0x7fd8640a7b90 2023-08-22T23:45:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.180+0000 7fd84bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.180+0000 7fd84bfff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:45:53.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.355+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 --> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd86413eca0 con 0x7fd85006b9e0 2023-08-22T23:45:53.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.361+0000 7fd84bfff640 1 -- 172.21.15.29:0/1196979284 <== mgr.4108 v2:172.21.15.52:6800/697719510 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152153 (secure 0 0 0) 0x7fd858001430 con 0x7fd85006b9e0 2023-08-22T23:45:53.363 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-08-22T23:45:53.364 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-08-22T23:45:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 10 librados: watch_flush enter 2023-08-22T23:45:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 10 librados: watch_flush exit 2023-08-22T23:45:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 20 client.16754.objecter shutdown clearing up homeless session... 2023-08-22T23:45:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 10 client.16754.objecter successfully canceled tick 2023-08-22T23:45:53.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd85006b9e0 msgr2=0x7fd85006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd85006b9e0 0x7fd85006dea0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd86413bbb0 tx=0x7fd858009040 comp rx=0 tx=0).stop 2023-08-22T23:45:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 10 monclient: shutdown 2023-08-22T23:45:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 msgr2=0x7fd864142710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 0x7fd864142710 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c530 tx=0x7fd85400c9f0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 shutdown_connections 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7fd85006b9e0 0x7fd85006dea0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8640b1380 0x7fd8641d9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8640a7f70 0x7fd8641d9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 --2- 172.21.15.29:0/1196979284 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7fd8640a7b90 0x7fd864142710 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 >> 172.21.15.29:0/1196979284 conn(0x7fd8640b6a00 msgr2=0x7fd86413e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 shutdown_connections 2023-08-22T23:45:53.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.363+0000 7fd869ccc640 1 -- 172.21.15.29:0/1196979284 wait complete. 2023-08-22T23:45:53.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.364+0000 7fd869ccc640 1 librados: shutdown 2023-08-22T23:45:53.375 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2023-08-22T23:45:52.590568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606456,"kb_used_data":100024,"kb_used_omap":13,"kb_used_meta":506354,"kb_avail":754368264,"statfs":{"total":773094113280,"available":772473102336,"internally_reserved":0,"allocated":102424576,"data_stored":62259580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":518506584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003059"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":876,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:56.865960+0000","last_change":"2023-08-22T23:44:56.867424+0000","last_active":"2023-08-22T23:44:56.865960+0000","last_peered":"2023-08-22T23:44:56.865960+0000","last_clean":"2023-08-22T23:44:56.865960+0000","last_became_active":"2023-08-22T23:23:40.385410+0000","last_became_peered":"2023-08-22T23:23:40.385410+0000","last_unstale":"2023-08-22T23:44:56.865960+0000","last_undegraded":"2023-08-22T23:44:56.865960+0000","last_fullsized":"2023-08-22T23:44:56.865960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:56.864518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:56.864518+0000","last_clean_scrub_stamp":"2023-08-22T23:44:56.864518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:47:28.870179+0000","scrub_duration":0.075777624000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":878,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:06.865880+0000","last_change":"2023-08-22T23:45:06.865880+0000","last_active":"2023-08-22T23:45:06.865880+0000","last_peered":"2023-08-22T23:45:06.865880+0000","last_clean":"2023-08-22T23:45:06.865880+0000","last_became_active":"2023-08-22T23:23:32.298152+0000","last_became_peered":"2023-08-22T23:23:32.298152+0000","last_unstale":"2023-08-22T23:45:06.865880+0000","last_undegraded":"2023-08-22T23:45:06.865880+0000","last_fullsized":"2023-08-22T23:45:06.865880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:06.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:06.860450+0000","last_clean_scrub_stamp":"2023-08-22T23:45:06.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:05:25.109783+0000","scrub_duration":0.052525585999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":890,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:31.662088+0000","last_change":"2023-08-22T23:45:31.667195+0000","last_active":"2023-08-22T23:45:31.662088+0000","last_peered":"2023-08-22T23:45:31.662088+0000","last_clean":"2023-08-22T23:45:31.662088+0000","last_became_active":"2023-08-22T23:23:41.235165+0000","last_became_peered":"2023-08-22T23:23:41.235165+0000","last_unstale":"2023-08-22T23:45:31.662088+0000","last_undegraded":"2023-08-22T23:45:31.662088+0000","last_fullsized":"2023-08-22T23:45:31.662088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:31.660044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:31.660044+0000","last_clean_scrub_stamp":"2023-08-22T23:45:31.660044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:30:53.203254+0000","scrub_duration":0.059672426000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.370589+0000","last_change":"2023-08-22T23:44:58.375301+0000","last_active":"2023-08-22T23:44:58.370589+0000","last_peered":"2023-08-22T23:44:58.370589+0000","last_clean":"2023-08-22T23:44:58.370589+0000","last_became_active":"2023-08-22T23:23:40.288979+0000","last_became_peered":"2023-08-22T23:23:40.288979+0000","last_unstale":"2023-08-22T23:44:58.370589+0000","last_undegraded":"2023-08-22T23:44:58.370589+0000","last_fullsized":"2023-08-22T23:44:58.370589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.369087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.369087+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.369087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:24:12.816396+0000","scrub_duration":0.031867338000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:01.069233+0000","last_change":"2023-08-22T23:45:01.070353+0000","last_active":"2023-08-22T23:45:01.069233+0000","last_peered":"2023-08-22T23:45:01.069233+0000","last_clean":"2023-08-22T23:45:01.069233+0000","last_became_active":"2023-08-22T23:23:35.294474+0000","last_became_peered":"2023-08-22T23:23:35.294474+0000","last_unstale":"2023-08-22T23:45:01.069233+0000","last_undegraded":"2023-08-22T23:45:01.069233+0000","last_fullsized":"2023-08-22T23:45:01.069233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:01.067805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:01.067805+0000","last_clean_scrub_stamp":"2023-08-22T23:45:01.067805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:57:44.262712+0000","scrub_duration":0.080561184999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:57.348370+0000","last_change":"2023-08-22T23:44:57.349504+0000","last_active":"2023-08-22T23:44:57.348370+0000","last_peered":"2023-08-22T23:44:57.348370+0000","last_clean":"2023-08-22T23:44:57.348370+0000","last_became_active":"2023-08-22T23:23:42.553380+0000","last_became_peered":"2023-08-22T23:23:42.553380+0000","last_unstale":"2023-08-22T23:44:57.348370+0000","last_undegraded":"2023-08-22T23:44:57.348370+0000","last_fullsized":"2023-08-22T23:44:57.348370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:57.346862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:57.346862+0000","last_clean_scrub_stamp":"2023-08-22T23:44:57.346862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:08:50.682485+0000","scrub_duration":0.036164746999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.037996+0000","last_change":"2023-08-22T23:44:58.037996+0000","last_active":"2023-08-22T23:44:58.037996+0000","last_peered":"2023-08-22T23:44:58.037996+0000","last_clean":"2023-08-22T23:44:58.037996+0000","last_became_active":"2023-08-22T23:23:39.784195+0000","last_became_peered":"2023-08-22T23:23:39.784195+0000","last_unstale":"2023-08-22T23:44:58.037996+0000","last_undegraded":"2023-08-22T23:44:58.037996+0000","last_fullsized":"2023-08-22T23:44:58.037996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.035322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.035322+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.035322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:02:25.891429+0000","scrub_duration":0.057636233000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":877,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.744778+0000","last_change":"2023-08-22T23:44:52.744778+0000","last_active":"2023-08-22T23:44:52.744778+0000","last_peered":"2023-08-22T23:44:52.744778+0000","last_clean":"2023-08-22T23:44:52.744778+0000","last_became_active":"2023-08-22T23:23:31.465434+0000","last_became_peered":"2023-08-22T23:23:31.465434+0000","last_unstale":"2023-08-22T23:44:52.744778+0000","last_undegraded":"2023-08-22T23:44:52.744778+0000","last_fullsized":"2023-08-22T23:44:52.744778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.738185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.738185+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.738185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:02:04.429354+0000","scrub_duration":0.031314258999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:01.868432+0000","last_change":"2023-08-22T23:45:01.868432+0000","last_active":"2023-08-22T23:45:01.868432+0000","last_peered":"2023-08-22T23:45:01.868432+0000","last_clean":"2023-08-22T23:45:01.868432+0000","last_became_active":"2023-08-22T23:23:30.864787+0000","last_became_peered":"2023-08-22T23:23:30.864787+0000","last_unstale":"2023-08-22T23:45:01.868432+0000","last_undegraded":"2023-08-22T23:45:01.868432+0000","last_fullsized":"2023-08-22T23:45:01.868432+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:01.865727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:01.865727+0000","last_clean_scrub_stamp":"2023-08-22T23:45:01.865727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:56:14.967946+0000","scrub_duration":0.082296422999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.674168+0000","last_change":"2023-08-22T23:44:53.674168+0000","last_active":"2023-08-22T23:44:53.674168+0000","last_peered":"2023-08-22T23:44:53.674168+0000","last_clean":"2023-08-22T23:44:53.674168+0000","last_became_active":"2023-08-22T23:23:57.352580+0000","last_became_peered":"2023-08-22T23:23:57.352580+0000","last_unstale":"2023-08-22T23:44:53.674168+0000","last_undegraded":"2023-08-22T23:44:53.674168+0000","last_fullsized":"2023-08-22T23:44:53.674168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.668393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.668393+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.668393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:07:44.060301+0000","scrub_duration":0.039868937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.542439+0000","last_change":"2023-08-22T23:44:48.542439+0000","last_active":"2023-08-22T23:44:48.542439+0000","last_peered":"2023-08-22T23:44:48.542439+0000","last_clean":"2023-08-22T23:44:48.542439+0000","last_became_active":"2023-08-22T23:23:36.209928+0000","last_became_peered":"2023-08-22T23:23:36.209928+0000","last_unstale":"2023-08-22T23:44:48.542439+0000","last_undegraded":"2023-08-22T23:44:48.542439+0000","last_fullsized":"2023-08-22T23:44:48.542439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.539506+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.539506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:42:32.540349+0000","scrub_duration":0.055977747000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:20.456307+0000","last_change":"2023-08-22T23:45:20.457468+0000","last_active":"2023-08-22T23:45:20.456307+0000","last_peered":"2023-08-22T23:45:20.456307+0000","last_clean":"2023-08-22T23:45:20.456307+0000","last_became_active":"2023-08-22T23:23:45.560294+0000","last_became_peered":"2023-08-22T23:23:45.560294+0000","last_unstale":"2023-08-22T23:45:20.456307+0000","last_undegraded":"2023-08-22T23:45:20.456307+0000","last_fullsized":"2023-08-22T23:45:20.456307+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:20.454769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:20.454769+0000","last_clean_scrub_stamp":"2023-08-22T23:45:20.454769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:03:05.855957+0000","scrub_duration":0.033051955000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":896,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:09.338967+0000","last_change":"2023-08-22T23:45:09.340155+0000","last_active":"2023-08-22T23:45:09.338967+0000","last_peered":"2023-08-22T23:45:09.338967+0000","last_clean":"2023-08-22T23:45:09.338967+0000","last_became_active":"2023-08-22T23:22:30.226473+0000","last_became_peered":"2023-08-22T23:22:30.226473+0000","last_unstale":"2023-08-22T23:45:09.338967+0000","last_undegraded":"2023-08-22T23:45:09.338967+0000","last_fullsized":"2023-08-22T23:45:09.338967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:09.337450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:09.337450+0000","last_clean_scrub_stamp":"2023-08-22T23:45:09.337450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:26:04.335737+0000","scrub_duration":0.031976879,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":885,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.037886+0000","last_change":"2023-08-22T23:44:52.037886+0000","last_active":"2023-08-22T23:44:52.037886+0000","last_peered":"2023-08-22T23:44:52.037886+0000","last_clean":"2023-08-22T23:44:52.037886+0000","last_became_active":"2023-08-22T23:23:37.080126+0000","last_became_peered":"2023-08-22T23:23:37.080126+0000","last_unstale":"2023-08-22T23:44:52.037886+0000","last_undegraded":"2023-08-22T23:44:52.037886+0000","last_fullsized":"2023-08-22T23:44:52.037886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.032042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.032042+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.032042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:04:20.437680+0000","scrub_duration":0.055160028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":889,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:48.903442+0000","last_change":"2023-08-22T23:44:48.907344+0000","last_active":"2023-08-22T23:44:48.903442+0000","last_peered":"2023-08-22T23:44:48.903442+0000","last_clean":"2023-08-22T23:44:48.903442+0000","last_became_active":"2023-08-22T23:22:30.247019+0000","last_became_peered":"2023-08-22T23:22:30.247019+0000","last_unstale":"2023-08-22T23:44:48.903442+0000","last_undegraded":"2023-08-22T23:44:48.903442+0000","last_fullsized":"2023-08-22T23:44:48.903442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:48.901951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:48.901951+0000","last_clean_scrub_stamp":"2023-08-22T23:44:48.901951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:56:14.326385+0000","scrub_duration":0.037514621999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":886,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.035238+0000","last_change":"2023-08-22T23:44:53.035238+0000","last_active":"2023-08-22T23:44:53.035238+0000","last_peered":"2023-08-22T23:44:53.035238+0000","last_clean":"2023-08-22T23:44:53.035238+0000","last_became_active":"2023-08-22T23:23:32.251158+0000","last_became_peered":"2023-08-22T23:23:32.251158+0000","last_unstale":"2023-08-22T23:44:53.035238+0000","last_undegraded":"2023-08-22T23:44:53.035238+0000","last_fullsized":"2023-08-22T23:44:53.035238+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.032673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.032673+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.032673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:32:17.339073+0000","scrub_duration":0.075927522999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:04.307187+0000","last_change":"2023-08-22T23:45:04.307187+0000","last_active":"2023-08-22T23:45:04.307187+0000","last_peered":"2023-08-22T23:45:04.307187+0000","last_clean":"2023-08-22T23:45:04.307187+0000","last_became_active":"2023-08-22T23:22:30.030812+0000","last_became_peered":"2023-08-22T23:22:30.030812+0000","last_unstale":"2023-08-22T23:45:04.307187+0000","last_undegraded":"2023-08-22T23:45:04.307187+0000","last_fullsized":"2023-08-22T23:45:04.307187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:04.304664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:04.304664+0000","last_clean_scrub_stamp":"2023-08-22T23:45:04.304664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T09:28:11.096423+0000","scrub_duration":0.036375984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":870,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:58.913093+0000","last_change":"2023-08-22T23:44:58.917636+0000","last_active":"2023-08-22T23:44:58.913093+0000","last_peered":"2023-08-22T23:44:58.913093+0000","last_clean":"2023-08-22T23:44:58.913093+0000","last_became_active":"2023-08-22T23:23:47.627504+0000","last_became_peered":"2023-08-22T23:23:47.627504+0000","last_unstale":"2023-08-22T23:44:58.913093+0000","last_undegraded":"2023-08-22T23:44:58.913093+0000","last_fullsized":"2023-08-22T23:44:58.913093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:58.911627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:58.911627+0000","last_clean_scrub_stamp":"2023-08-22T23:44:58.911627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:12:11.522411+0000","scrub_duration":0.032531514999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":881,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:50.739077+0000","last_change":"2023-08-22T23:44:50.739077+0000","last_active":"2023-08-22T23:44:50.739077+0000","last_peered":"2023-08-22T23:44:50.739077+0000","last_clean":"2023-08-22T23:44:50.739077+0000","last_became_active":"2023-08-22T23:22:30.415331+0000","last_became_peered":"2023-08-22T23:22:30.415331+0000","last_unstale":"2023-08-22T23:44:50.739077+0000","last_undegraded":"2023-08-22T23:44:50.739077+0000","last_fullsized":"2023-08-22T23:44:50.739077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:50.736264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:50.736264+0000","last_clean_scrub_stamp":"2023-08-22T23:44:50.736264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:58:36.038766+0000","scrub_duration":0.044869521000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:47.828710+0000","last_change":"2023-08-22T23:44:47.829841+0000","last_active":"2023-08-22T23:44:47.828710+0000","last_peered":"2023-08-22T23:44:47.828710+0000","last_clean":"2023-08-22T23:44:47.828710+0000","last_became_active":"2023-08-22T23:23:44.746561+0000","last_became_peered":"2023-08-22T23:23:44.746561+0000","last_unstale":"2023-08-22T23:44:47.828710+0000","last_undegraded":"2023-08-22T23:44:47.828710+0000","last_fullsized":"2023-08-22T23:44:47.828710+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:47.827283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:47.827283+0000","last_clean_scrub_stamp":"2023-08-22T23:44:47.827283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T11:01:52.954243+0000","scrub_duration":0.069352237999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":875,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.638075+0000","last_change":"2023-08-22T23:44:51.639230+0000","last_active":"2023-08-22T23:44:51.638075+0000","last_peered":"2023-08-22T23:44:51.638075+0000","last_clean":"2023-08-22T23:44:51.638075+0000","last_became_active":"2023-08-22T23:23:43.791090+0000","last_became_peered":"2023-08-22T23:23:43.791090+0000","last_unstale":"2023-08-22T23:44:51.638075+0000","last_undegraded":"2023-08-22T23:44:51.638075+0000","last_fullsized":"2023-08-22T23:44:51.638075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:51.636674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:51.636674+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.636674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:06:31.966373+0000","scrub_duration":0.052995683000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.072250+0000","last_change":"2023-08-22T23:44:51.072250+0000","last_active":"2023-08-22T23:44:51.072250+0000","last_peered":"2023-08-22T23:44:51.072250+0000","last_clean":"2023-08-22T23:44:51.072250+0000","last_became_active":"2023-08-22T23:22:30.387497+0000","last_became_peered":"2023-08-22T23:22:30.387497+0000","last_unstale":"2023-08-22T23:44:51.072250+0000","last_undegraded":"2023-08-22T23:44:51.072250+0000","last_fullsized":"2023-08-22T23:44:51.072250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:51.066863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:51.066863+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.066863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:51:38.875911+0000","scrub_duration":0.038690479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.795936+0000","last_change":"2023-08-22T23:44:54.795936+0000","last_active":"2023-08-22T23:44:54.795936+0000","last_peered":"2023-08-22T23:44:54.795936+0000","last_clean":"2023-08-22T23:44:54.795936+0000","last_became_active":"2023-08-22T23:23:31.290119+0000","last_became_peered":"2023-08-22T23:23:31.290119+0000","last_unstale":"2023-08-22T23:44:54.795936+0000","last_undegraded":"2023-08-22T23:44:54.795936+0000","last_fullsized":"2023-08-22T23:44:54.795936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.792995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.792995+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.792995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:03:09.502831+0000","scrub_duration":0.033049222000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":890,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.481421+0000","last_change":"2023-08-22T23:44:52.481421+0000","last_active":"2023-08-22T23:44:52.481421+0000","last_peered":"2023-08-22T23:44:52.481421+0000","last_clean":"2023-08-22T23:44:52.481421+0000","last_became_active":"2023-08-22T23:22:31.548165+0000","last_became_peered":"2023-08-22T23:22:31.548165+0000","last_unstale":"2023-08-22T23:44:52.481421+0000","last_undegraded":"2023-08-22T23:44:52.481421+0000","last_fullsized":"2023-08-22T23:44:52.481421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.478869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.478869+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.478869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:23:01.134961+0000","scrub_duration":0.031767374000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"918'8","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.900114+0000","last_change":"2023-08-22T23:44:52.901399+0000","last_active":"2023-08-22T23:44:52.900114+0000","last_peered":"2023-08-22T23:44:52.900114+0000","last_clean":"2023-08-22T23:44:52.900114+0000","last_became_active":"2023-08-22T23:23:56.011321+0000","last_became_peered":"2023-08-22T23:23:56.011321+0000","last_unstale":"2023-08-22T23:44:52.900114+0000","last_undegraded":"2023-08-22T23:44:52.900114+0000","last_fullsized":"2023-08-22T23:44:52.900114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8","last_scrub_stamp":"2023-08-22T23:44:52.898702+0000","last_deep_scrub":"918'8","last_deep_scrub_stamp":"2023-08-22T23:44:52.898702+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.898702+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:50:16.475293+0000","scrub_duration":0.076325736000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:50.238325+0000","last_change":"2023-08-22T23:44:50.250441+0000","last_active":"2023-08-22T23:44:50.238325+0000","last_peered":"2023-08-22T23:44:50.238325+0000","last_clean":"2023-08-22T23:44:50.238325+0000","last_became_active":"2023-08-22T23:22:29.939068+0000","last_became_peered":"2023-08-22T23:22:29.939068+0000","last_unstale":"2023-08-22T23:44:50.238325+0000","last_undegraded":"2023-08-22T23:44:50.238325+0000","last_fullsized":"2023-08-22T23:44:50.238325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:50.229791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:50.229791+0000","last_clean_scrub_stamp":"2023-08-22T23:44:50.229791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:58:24.193218+0000","scrub_duration":0.21635689899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.008748+0000","last_change":"2023-08-22T23:44:54.009903+0000","last_active":"2023-08-22T23:44:54.008748+0000","last_peered":"2023-08-22T23:44:54.008748+0000","last_clean":"2023-08-22T23:44:54.008748+0000","last_became_active":"2023-08-22T23:23:34.100153+0000","last_became_peered":"2023-08-22T23:23:34.100153+0000","last_unstale":"2023-08-22T23:44:54.008748+0000","last_undegraded":"2023-08-22T23:44:54.008748+0000","last_fullsized":"2023-08-22T23:44:54.008748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.007311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.007311+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.007311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:49:21.630484+0000","scrub_duration":0.043985337999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":897,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:59.285871+0000","last_change":"2023-08-22T23:44:59.285871+0000","last_active":"2023-08-22T23:44:59.285871+0000","last_peered":"2023-08-22T23:44:59.285871+0000","last_clean":"2023-08-22T23:44:59.285871+0000","last_became_active":"2023-08-22T23:22:30.323880+0000","last_became_peered":"2023-08-22T23:22:30.323880+0000","last_unstale":"2023-08-22T23:44:59.285871+0000","last_undegraded":"2023-08-22T23:44:59.285871+0000","last_fullsized":"2023-08-22T23:44:59.285871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:59.283191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:59.283191+0000","last_clean_scrub_stamp":"2023-08-22T23:44:59.283191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T10:16:55.698349+0000","scrub_duration":0.032512915000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"918'78","reported_seq":995,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:51.503440+0000","last_change":"2023-08-22T23:44:51.507885+0000","last_active":"2023-08-22T23:44:51.503440+0000","last_peered":"2023-08-22T23:44:51.503440+0000","last_clean":"2023-08-22T23:44:51.503440+0000","last_became_active":"2023-08-22T23:22:24.309401+0000","last_became_peered":"2023-08-22T23:22:24.309401+0000","last_unstale":"2023-08-22T23:44:51.503440+0000","last_undegraded":"2023-08-22T23:44:51.503440+0000","last_fullsized":"2023-08-22T23:44:51.503440+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'78","last_scrub_stamp":"2023-08-22T23:44:51.501949+0000","last_deep_scrub":"918'78","last_deep_scrub_stamp":"2023-08-22T23:44:51.501949+0000","last_clean_scrub_stamp":"2023-08-22T23:44:51.501949+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T05:17:02.406666+0000","scrub_duration":0.125489981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":887,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:00.062115+0000","last_change":"2023-08-22T23:45:00.062115+0000","last_active":"2023-08-22T23:45:00.062115+0000","last_peered":"2023-08-22T23:45:00.062115+0000","last_clean":"2023-08-22T23:45:00.062115+0000","last_became_active":"2023-08-22T23:23:37.115129+0000","last_became_peered":"2023-08-22T23:23:37.115129+0000","last_unstale":"2023-08-22T23:45:00.062115+0000","last_undegraded":"2023-08-22T23:45:00.062115+0000","last_fullsized":"2023-08-22T23:45:00.062115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:00.059445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:00.059445+0000","last_clean_scrub_stamp":"2023-08-22T23:45:00.059445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:10:12.317174+0000","scrub_duration":0.060663304000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:11.450240+0000","last_change":"2023-08-22T23:45:11.450240+0000","last_active":"2023-08-22T23:45:11.450240+0000","last_peered":"2023-08-22T23:45:11.450240+0000","last_clean":"2023-08-22T23:45:11.450240+0000","last_became_active":"2023-08-22T23:23:33.754756+0000","last_became_peered":"2023-08-22T23:23:33.754756+0000","last_unstale":"2023-08-22T23:45:11.450240+0000","last_undegraded":"2023-08-22T23:45:11.450240+0000","last_fullsized":"2023-08-22T23:45:11.450240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:11.444179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:11.444179+0000","last_clean_scrub_stamp":"2023-08-22T23:45:11.444179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T07:06:04.051365+0000","scrub_duration":0.045936974999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":892,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:17.402867+0000","last_change":"2023-08-22T23:45:17.402867+0000","last_active":"2023-08-22T23:45:17.402867+0000","last_peered":"2023-08-22T23:45:17.402867+0000","last_clean":"2023-08-22T23:45:17.402867+0000","last_became_active":"2023-08-22T23:23:32.980094+0000","last_became_peered":"2023-08-22T23:23:32.980094+0000","last_unstale":"2023-08-22T23:45:17.402867+0000","last_undegraded":"2023-08-22T23:45:17.402867+0000","last_fullsized":"2023-08-22T23:45:17.402867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:17.400248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:17.400248+0000","last_clean_scrub_stamp":"2023-08-22T23:45:17.400248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:47:16.279822+0000","scrub_duration":0.042204931000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.573413+0000","last_change":"2023-08-22T23:44:54.574547+0000","last_active":"2023-08-22T23:44:54.573413+0000","last_peered":"2023-08-22T23:44:54.573413+0000","last_clean":"2023-08-22T23:44:54.573413+0000","last_became_active":"2023-08-22T23:23:36.044555+0000","last_became_peered":"2023-08-22T23:23:36.044555+0000","last_unstale":"2023-08-22T23:44:54.573413+0000","last_undegraded":"2023-08-22T23:44:54.573413+0000","last_fullsized":"2023-08-22T23:44:54.573413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.571987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.571987+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.571987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T00:24:26.605250+0000","scrub_duration":0.091127818999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:46.514602+0000","last_change":"2023-08-22T23:44:46.515850+0000","last_active":"2023-08-22T23:44:46.514602+0000","last_peered":"2023-08-22T23:44:46.514602+0000","last_clean":"2023-08-22T23:44:46.514602+0000","last_became_active":"2023-08-22T23:23:32.320752+0000","last_became_peered":"2023-08-22T23:23:32.320752+0000","last_unstale":"2023-08-22T23:44:46.514602+0000","last_undegraded":"2023-08-22T23:44:46.514602+0000","last_fullsized":"2023-08-22T23:44:46.514602+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:46.513081+0000","last_clean_scrub_stamp":"2023-08-22T23:44:46.513081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T08:49:45.949593+0000","scrub_duration":0.047647529000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:26.570329+0000","last_change":"2023-08-22T23:45:26.571547+0000","last_active":"2023-08-22T23:45:26.570329+0000","last_peered":"2023-08-22T23:45:26.570329+0000","last_clean":"2023-08-22T23:45:26.570329+0000","last_became_active":"2023-08-22T23:23:32.889371+0000","last_became_peered":"2023-08-22T23:23:32.889371+0000","last_unstale":"2023-08-22T23:45:26.570329+0000","last_undegraded":"2023-08-22T23:45:26.570329+0000","last_fullsized":"2023-08-22T23:45:26.570329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:26.568781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:26.568781+0000","last_clean_scrub_stamp":"2023-08-22T23:45:26.568781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T04:58:57.463721+0000","scrub_duration":0.051628172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:52.673754+0000","last_change":"2023-08-22T23:44:52.674889+0000","last_active":"2023-08-22T23:44:52.673754+0000","last_peered":"2023-08-22T23:44:52.673754+0000","last_clean":"2023-08-22T23:44:52.673754+0000","last_became_active":"2023-08-22T23:23:55.258705+0000","last_became_peered":"2023-08-22T23:23:55.258705+0000","last_unstale":"2023-08-22T23:44:52.673754+0000","last_undegraded":"2023-08-22T23:44:52.673754+0000","last_fullsized":"2023-08-22T23:44:52.673754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:52.672298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:52.672298+0000","last_clean_scrub_stamp":"2023-08-22T23:44:52.672298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T06:59:07.685804+0000","scrub_duration":0.032147282999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":874,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:49.599748+0000","last_change":"2023-08-22T23:44:49.600902+0000","last_active":"2023-08-22T23:44:49.599748+0000","last_peered":"2023-08-22T23:44:49.599748+0000","last_clean":"2023-08-22T23:44:49.599748+0000","last_became_active":"2023-08-22T23:23:39.702079+0000","last_became_peered":"2023-08-22T23:23:39.702079+0000","last_unstale":"2023-08-22T23:44:49.599748+0000","last_undegraded":"2023-08-22T23:44:49.599748+0000","last_fullsized":"2023-08-22T23:44:49.599748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:49.598301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:49.598301+0000","last_clean_scrub_stamp":"2023-08-22T23:44:49.598301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:13:36.863120+0000","scrub_duration":0.084724123999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:53.377506+0000","last_change":"2023-08-22T23:44:53.378666+0000","last_active":"2023-08-22T23:44:53.377506+0000","last_peered":"2023-08-22T23:44:53.377506+0000","last_clean":"2023-08-22T23:44:53.377506+0000","last_became_active":"2023-08-22T23:23:43.437692+0000","last_became_peered":"2023-08-22T23:23:43.437692+0000","last_unstale":"2023-08-22T23:44:53.377506+0000","last_undegraded":"2023-08-22T23:44:53.377506+0000","last_fullsized":"2023-08-22T23:44:53.377506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:53.376022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:53.376022+0000","last_clean_scrub_stamp":"2023-08-22T23:44:53.376022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:07:11.775246+0000","scrub_duration":0.036018521999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:16.430482+0000","last_change":"2023-08-22T23:45:16.435059+0000","last_active":"2023-08-22T23:45:16.430482+0000","last_peered":"2023-08-22T23:45:16.430482+0000","last_clean":"2023-08-22T23:45:16.430482+0000","last_became_active":"2023-08-22T23:23:45.500786+0000","last_became_peered":"2023-08-22T23:23:45.500786+0000","last_unstale":"2023-08-22T23:45:16.430482+0000","last_undegraded":"2023-08-22T23:45:16.430482+0000","last_fullsized":"2023-08-22T23:45:16.430482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:16.428969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:16.428969+0000","last_clean_scrub_stamp":"2023-08-22T23:45:16.428969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T03:15:17.601180+0000","scrub_duration":0.054305531999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":869,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:44:54.896115+0000","last_change":"2023-08-22T23:44:54.897241+0000","last_active":"2023-08-22T23:44:54.896115+0000","last_peered":"2023-08-22T23:44:54.896115+0000","last_clean":"2023-08-22T23:44:54.896115+0000","last_became_active":"2023-08-22T23:23:33.840895+0000","last_became_peered":"2023-08-22T23:23:33.840895+0000","last_unstale":"2023-08-22T23:44:54.896115+0000","last_undegraded":"2023-08-22T23:44:54.896115+0000","last_fullsized":"2023-08-22T23:44:54.896115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:44:54.892207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:44:54.892207+0000","last_clean_scrub_stamp":"2023-08-22T23:44:54.892207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T01:22:01.531638+0000","scrub_duration":0.03351606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":891,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-08-22T23:45:18.444771+0000","last_change":"2023-08-22T23:45:18.445952+0000","last_active":"2023-08-22T23:45:18.444771+0000","last_peered":"2023-08-22T23:45:18.444771+0000","last_clean":"2023-08-22T23:45:18.444771+0000","last_became_active":"2023-08-22T23:23:41.089191+0000","last_became_peered":"2023-08-22T23:23:41.089191+0000","last_unstale":"2023-08-22T23:45:18.444771+0000","last_undegraded":"2023-08-22T23:45:18.444771+0000","last_fullsized":"2023-08-22T23:45:18.444771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-22T23:45:18.443189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-22T23:45:18.443189+0000","last_clean_scrub_stamp":"2023-08-22T23:45:18.443189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-08-24T02:21:33.407169+0000","scrub_duration":0.075346278000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":16,"seq":68719477019,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65888,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305952,"statfs":{"total":96636764160,"available":96569294848,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":19.638999999999999,"5min":16.050999999999998,"15min":18.748999999999999},"min":{"1min":7.3460000000000001,"5min":4.734,"15min":4.734},"max":{"1min":120.896,"5min":120.896,"15min":254.63},"last":8.0640000000000001},{"interface":"front","average":{"1min":21.446999999999999,"5min":19.369,"15min":21.602},"min":{"1min":6.3570000000000002,"5min":4.9249999999999998,"15min":4.9249999999999998},"max":{"1min":127.82299999999999,"5min":127.82299999999999,"15min":330.00099999999998},"last":10.912000000000001}]},{"osd":2,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":18.699000000000002,"5min":19.863,"15min":20.460000000000001},"min":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":6.1749999999999998},"max":{"1min":96.775000000000006,"5min":175.85499999999999,"15min":176.803},"last":9.968},{"interface":"front","average":{"1min":22.876000000000001,"5min":24.242000000000001,"15min":23.904},"min":{"1min":7.2439999999999998,"5min":7.0099999999999998,"15min":6.9729999999999999},"max":{"1min":126.717,"5min":267.399,"15min":277.048},"last":11.571999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":18.198,"5min":16.169,"15min":18.484999999999999},"min":{"1min":6.0919999999999996,"5min":6.0919999999999996,"15min":5.766},"max":{"1min":106.98699999999999,"5min":112.148,"15min":332.09199999999998},"last":7.4820000000000002},{"interface":"front","average":{"1min":26.771999999999998,"5min":21.484999999999999,"15min":24.721},"min":{"1min":9.8439999999999994,"5min":7.0739999999999998,"15min":7.0739999999999998},"max":{"1min":221.70500000000001,"5min":221.70500000000001,"15min":221.70500000000001},"last":17.928000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":17.216000000000001,"5min":19.616,"15min":23.817},"min":{"1min":9.3539999999999992,"5min":7.5789999999999997,"15min":7.4969999999999999},"max":{"1min":71.573999999999998,"5min":107.697,"15min":333.517},"last":15.144},{"interface":"front","average":{"1min":18.596,"5min":23.498000000000001,"15min":27.713000000000001},"min":{"1min":9.5150000000000006,"5min":9.5150000000000006,"15min":8.5779999999999994},"max":{"1min":67.555999999999997,"5min":126.54000000000001,"15min":342.31999999999999},"last":12.295}]},{"osd":5,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":23.25,"5min":21.058,"15min":22.721},"min":{"1min":9.6099999999999994,"5min":7.1609999999999996,"15min":6.0869999999999997},"max":{"1min":81.114000000000004,"5min":104.696,"15min":326.71300000000002},"last":12.923},{"interface":"front","average":{"1min":20.375,"5min":19.539000000000001,"15min":25.965},"min":{"1min":7.1680000000000001,"5min":7.1680000000000001,"15min":7.1680000000000001},"max":{"1min":85.491,"5min":99.808000000000007,"15min":250.43600000000001},"last":8.7279999999999998}]},{"osd":6,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":22.710999999999999,"5min":20.698,"15min":25.279},"min":{"1min":11.891999999999999,"5min":9.5229999999999997,"15min":8.9860000000000007},"max":{"1min":82.162000000000006,"5min":82.162000000000006,"15min":267.84800000000001},"last":16.599},{"interface":"front","average":{"1min":27.213999999999999,"5min":26.039000000000001,"15min":28.754000000000001},"min":{"1min":12.481,"5min":10.958,"15min":9.8079999999999998},"max":{"1min":104.075,"5min":105.946,"15min":255.58699999999999},"last":13.592000000000001}]},{"osd":7,"last update":"Tue Aug 22 23:45:08 2023","interfaces":[{"interface":"back","average":{"1min":25.402000000000001,"5min":27.661999999999999,"15min":31.119},"min":{"1min":12.215999999999999,"5min":11.016,"15min":10.459},"max":{"1min":76.296999999999997,"5min":146.363,"15min":274.60899999999998},"last":17.279},{"interface":"front","average":{"1min":18.678999999999998,"5min":20.068000000000001,"15min":22.187999999999999},"min":{"1min":8.4269999999999996,"5min":8.4269999999999996,"15min":7.8300000000000001},"max":{"1min":84.183999999999997,"5min":114.447,"15min":194.761},"last":15.846}]}]},{"osd":3,"up_from":16,"seq":68719477020,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83544,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94288296,"statfs":{"total":96636764160,"available":96551215104,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":15.458,"5min":16.449999999999999,"15min":18.245999999999999},"min":{"1min":6.625,"5min":5.6189999999999998,"15min":5.0359999999999996},"max":{"1min":60.25,"5min":153.98599999999999,"15min":220.95400000000001},"last":8.8070000000000004},{"interface":"front","average":{"1min":20.081,"5min":23.474,"15min":25.812000000000001},"min":{"1min":10.656000000000001,"5min":9.2579999999999991,"15min":6.3620000000000001},"max":{"1min":65.772000000000006,"5min":215.34800000000001,"15min":356.75599999999997},"last":17.231000000000002}]},{"osd":1,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":13.866,"5min":16.361000000000001,"15min":19.143000000000001},"min":{"1min":6.2240000000000002,"5min":5.218,"15min":5.141},"max":{"1min":59.353999999999999,"5min":189.31800000000001,"15min":359.32999999999998},"last":9.8710000000000004},{"interface":"front","average":{"1min":14.013,"5min":16.992000000000001,"15min":19.372},"min":{"1min":8.0760000000000005,"5min":5.9550000000000001,"15min":5.9480000000000004},"max":{"1min":63.573,"5min":190.49799999999999,"15min":374.01799999999997},"last":8.0530000000000008}]},{"osd":2,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":17.425000000000001,"5min":22.100000000000001,"15min":21.399000000000001},"min":{"1min":8.0069999999999997,"5min":6.524,"15min":6.524},"max":{"1min":64.561000000000007,"5min":198.649,"15min":233.745},"last":15.112},{"interface":"front","average":{"1min":17.495999999999999,"5min":20.888000000000002,"15min":22.417999999999999},"min":{"1min":7.8179999999999996,"5min":5.7720000000000002,"15min":5.7720000000000002},"max":{"1min":70.597999999999999,"5min":193.53999999999999,"15min":241.23500000000001},"last":14.483000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":17.477,"5min":21.974,"15min":27.140000000000001},"min":{"1min":8.7720000000000002,"5min":8.7720000000000002,"15min":6.6230000000000002},"max":{"1min":72.275000000000006,"5min":199.42400000000001,"15min":370.84399999999999},"last":20.125},{"interface":"front","average":{"1min":19.637,"5min":24.417000000000002,"15min":30.408999999999999},"min":{"1min":8.4450000000000003,"5min":6.726,"15min":6.726},"max":{"1min":89.596999999999994,"5min":231.53,"15min":372.45299999999997},"last":13.07}]},{"osd":5,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":21.088999999999999,"15min":24.824999999999999},"min":{"1min":9.6869999999999994,"5min":9.6869999999999994,"15min":8.5419999999999998},"max":{"1min":67.260999999999996,"5min":175.429,"15min":241.86099999999999},"last":13.741},{"interface":"front","average":{"1min":18.904,"5min":20.405000000000001,"15min":24.536000000000001},"min":{"1min":8.6530000000000005,"5min":7.7110000000000003,"15min":6.0540000000000003},"max":{"1min":74.754999999999995,"5min":212.09,"15min":289.34300000000002},"last":17.821000000000002}]},{"osd":6,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":20.495000000000001,"5min":22.859999999999999,"15min":27.015999999999998},"min":{"1min":11.388999999999999,"5min":8.8979999999999997,"15min":8.8979999999999997},"max":{"1min":68.186000000000007,"5min":213.15000000000001,"15min":246.30000000000001},"last":10.816000000000001},{"interface":"front","average":{"1min":25.173999999999999,"5min":28.638000000000002,"15min":27.481999999999999},"min":{"1min":12.978,"5min":8.1669999999999998,"15min":7.7830000000000004},"max":{"1min":73.796000000000006,"5min":215.41300000000001,"15min":215.41300000000001},"last":18.548999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":22.321999999999999,"5min":22.753,"15min":28.486000000000001},"min":{"1min":12.140000000000001,"5min":8.3369999999999997,"15min":7.7439999999999998},"max":{"1min":76.263999999999996,"5min":206.708,"15min":417.863},"last":12.242000000000001},{"interface":"front","average":{"1min":22.802,"5min":27.838999999999999,"15min":28.358000000000001},"min":{"1min":13.613,"5min":11.994999999999999,"15min":8.0549999999999997},"max":{"1min":71.433999999999997,"5min":229.85900000000001,"15min":229.85900000000001},"last":19.170000000000002}]}]},{"osd":2,"up_from":16,"seq":68719477019,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65500,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94306340,"statfs":{"total":96636764160,"available":96569692160,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":10.722,"5min":15.116,"15min":16.167999999999999},"min":{"1min":6.5270000000000001,"5min":5.9349999999999996,"15min":4.8319999999999999},"max":{"1min":25.141999999999999,"5min":204.17500000000001,"15min":208.114},"last":14.435},{"interface":"front","average":{"1min":15.356999999999999,"5min":20.913,"15min":19.713000000000001},"min":{"1min":9.109,"5min":5.9450000000000003,"15min":5.7089999999999996},"max":{"1min":31.975000000000001,"5min":205.833,"15min":205.833},"last":12.945}]},{"osd":1,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":12.912000000000001,"5min":16.635999999999999,"15min":17.698},"min":{"1min":6.9400000000000004,"5min":6.6589999999999998,"15min":6.2380000000000004},"max":{"1min":57.646000000000001,"5min":142.42599999999999,"15min":218.06200000000001},"last":15.944000000000001},{"interface":"front","average":{"1min":19.082999999999998,"5min":21.914000000000001,"15min":21.427},"min":{"1min":10.345000000000001,"5min":10.121,"15min":7.0250000000000004},"max":{"1min":62.720999999999997,"5min":145.83000000000001,"15min":230.108},"last":18.123999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:44:54 2023","interfaces":[{"interface":"back","average":{"1min":11.27,"5min":21.934999999999999,"15min":20.477},"min":{"1min":6.9749999999999996,"5min":6.3310000000000004,"15min":6.3310000000000004},"max":{"1min":23.353999999999999,"5min":292.49000000000001,"15min":292.49000000000001},"last":8.1359999999999992},{"interface":"front","average":{"1min":20.030000000000001,"5min":28.835999999999999,"15min":25.010999999999999},"min":{"1min":10.519,"5min":8.6639999999999997,"15min":6.056},"max":{"1min":65.405000000000001,"5min":304.017,"15min":306.26299999999998},"last":11.468}]},{"osd":4,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":15.063000000000001,"5min":19.024999999999999,"15min":20.800000000000001},"min":{"1min":9.0210000000000008,"5min":8.7170000000000005,"15min":7.3869999999999996},"max":{"1min":33.100999999999999,"5min":157.559,"15min":187.357},"last":17.202999999999999},{"interface":"front","average":{"1min":15.558,"5min":18.388999999999999,"15min":20.344000000000001},"min":{"1min":8.9309999999999992,"5min":7.5830000000000002,"15min":6.9320000000000004},"max":{"1min":21.797999999999998,"5min":146.64599999999999,"15min":206.59100000000001},"last":15.138999999999999}]},{"osd":5,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":18.018000000000001,"5min":22.248000000000001,"15min":23.747},"min":{"1min":11.853,"5min":8.9640000000000004,"15min":8.8019999999999996},"max":{"1min":33.109000000000002,"5min":198.34200000000001,"15min":198.34200000000001},"last":19.02},{"interface":"front","average":{"1min":16.687000000000001,"5min":21.004000000000001,"15min":22.106000000000002},"min":{"1min":8.234,"5min":8.093,"15min":7.7629999999999999},"max":{"1min":30.888000000000002,"5min":181.24199999999999,"15min":181.24199999999999},"last":10.803000000000001}]},{"osd":6,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":17.91,"5min":24.329000000000001,"15min":23.096},"min":{"1min":11.965999999999999,"5min":11.346,"15min":8.6539999999999999},"max":{"1min":30.242000000000001,"5min":189.55000000000001,"15min":189.55000000000001},"last":20.047000000000001},{"interface":"front","average":{"1min":17.471,"5min":21.184000000000001,"15min":22.276},"min":{"1min":9.7170000000000005,"5min":9.2210000000000001,"15min":8.8819999999999997},"max":{"1min":29.152999999999999,"5min":129.928,"15min":130.45599999999999},"last":12.173999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:45:16 2023","interfaces":[{"interface":"back","average":{"1min":23.861999999999998,"5min":26.873000000000001,"15min":28.923999999999999},"min":{"1min":11.734,"5min":11.529999999999999,"15min":9.9939999999999998},"max":{"1min":67.033000000000001,"5min":195.655,"15min":402.06099999999998},"last":20.981000000000002},{"interface":"front","average":{"1min":16.623999999999999,"5min":21.030000000000001,"15min":27.128},"min":{"1min":7.8840000000000003,"5min":7.8840000000000003,"15min":7.8840000000000003},"max":{"1min":34.084000000000003,"5min":158.26499999999999,"15min":427.274},"last":9.6690000000000005}]}]},{"osd":0,"up_from":16,"seq":68719477020,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66152,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94305688,"statfs":{"total":96636764160,"available":96569024512,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":11.734,"5min":13.788,"15min":15.262},"min":{"1min":7.1319999999999997,"5min":6.1239999999999997,"15min":5.9939999999999998},"max":{"1min":30.100999999999999,"5min":59.094999999999999,"15min":164.69999999999999},"last":10.459},{"interface":"front","average":{"1min":12.417,"5min":17.076000000000001,"15min":17.960999999999999},"min":{"1min":7.431,"5min":5.7839999999999998,"15min":5.7839999999999998},"max":{"1min":28.806000000000001,"5min":150.68199999999999,"15min":256.70100000000002},"last":18.045999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":13.44,"5min":18.806000000000001,"15min":20.181999999999999},"min":{"1min":9.2899999999999991,"5min":7.5519999999999996,"15min":6.7830000000000004},"max":{"1min":19.698,"5min":133.678,"15min":255.041},"last":19.917999999999999},{"interface":"front","average":{"1min":12.794,"5min":18.503,"15min":20.234000000000002},"min":{"1min":8.4459999999999997,"5min":6.585,"15min":5.6639999999999997},"max":{"1min":23.007999999999999,"5min":122.78700000000001,"15min":266.08600000000001},"last":13.147}]},{"osd":3,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":10.009,"5min":18.423999999999999,"15min":18.460999999999999},"min":{"1min":6.7750000000000004,"5min":6.7050000000000001,"15min":5.3399999999999999},"max":{"1min":16.966999999999999,"5min":140.357,"15min":269.73200000000003},"last":7.5529999999999999},{"interface":"front","average":{"1min":16.195,"5min":20.917999999999999,"15min":21.375},"min":{"1min":10.243,"5min":6.8760000000000003,"15min":6.8760000000000003},"max":{"1min":29.484000000000002,"5min":124.69499999999999,"15min":253.67099999999999},"last":18.771999999999998}]},{"osd":4,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":13.577999999999999,"5min":17.079999999999998,"15min":21.831},"min":{"1min":7.3380000000000001,"5min":6.4939999999999998,"15min":6.4939999999999998},"max":{"1min":19.238,"5min":101.029,"15min":270.06900000000002},"last":13.826000000000001},{"interface":"front","average":{"1min":12.122,"5min":19.228000000000002,"15min":22.085000000000001},"min":{"1min":6.9649999999999999,"5min":6.9649999999999999,"15min":6.9649999999999999},"max":{"1min":19.108000000000001,"5min":137.608,"15min":254.87899999999999},"last":9.7240000000000002}]},{"osd":5,"last update":"Tue Aug 22 23:45:04 2023","interfaces":[{"interface":"back","average":{"1min":27.023,"5min":25.725999999999999,"15min":25.164000000000001},"min":{"1min":8.9079999999999995,"5min":7.7199999999999998,"15min":7.4100000000000001},"max":{"1min":263.81599999999997,"5min":263.81599999999997,"15min":287.80200000000002},"last":12.281000000000001},{"interface":"front","average":{"1min":29.484999999999999,"5min":25.931000000000001,"15min":26.088000000000001},"min":{"1min":10.371,"5min":8.5920000000000005,"15min":7.8460000000000001},"max":{"1min":272.935,"5min":272.935,"15min":272.935},"last":20.640999999999998}]},{"osd":6,"last update":"Tue Aug 22 23:45:22 2023","interfaces":[{"interface":"back","average":{"1min":18.088000000000001,"5min":25.870999999999999,"15min":24.484000000000002},"min":{"1min":13.003,"5min":10.363,"15min":7.0369999999999999},"max":{"1min":28.029,"5min":167.541,"15min":276.935},"last":21.809999999999999},{"interface":"front","average":{"1min":15.387,"5min":20.478000000000002,"15min":23.152000000000001},"min":{"1min":7.8179999999999996,"5min":7.8179999999999996,"15min":7.8179999999999996},"max":{"1min":26.350999999999999,"5min":109.886,"15min":265.17399999999998},"last":14.544}]},{"osd":7,"last update":"Tue Aug 22 23:45:04 2023","interfaces":[{"interface":"back","average":{"1min":18.571000000000002,"5min":27.088999999999999,"15min":30.423999999999999},"min":{"1min":14.311999999999999,"5min":10.805999999999999,"15min":7.1760000000000002},"max":{"1min":32.917000000000002,"5min":184.333,"15min":770.02200000000005},"last":15.303000000000001},{"interface":"front","average":{"1min":15.192,"5min":24.359999999999999,"15min":30.084},"min":{"1min":10.214,"5min":9.5999999999999996,"15min":8.0239999999999991},"max":{"1min":21.288,"5min":154.00800000000001,"15min":834.71799999999996},"last":16.109999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509725,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65500,"kb_used_data":12244,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94306340,"statfs":{"total":96636764160,"available":96569692160,"internally_reserved":0,"allocated":12537856,"data_stored":7553014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":54524171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":15.823,"5min":15.853999999999999,"15min":16.640999999999998},"min":{"1min":5.774,"5min":5.774,"15min":4.5570000000000004},"max":{"1min":77.439999999999998,"5min":101.271,"15min":190.38800000000001},"last":10.816000000000001},{"interface":"front","average":{"1min":25.606000000000002,"5min":23.318999999999999,"15min":21.681000000000001},"min":{"1min":9.0399999999999991,"5min":9.0399999999999991,"15min":4.9699999999999998},"max":{"1min":120.398,"5min":122.059,"15min":215.38800000000001},"last":13.368}]},{"osd":1,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":18.706,"5min":15.819000000000001,"15min":16.527999999999999},"min":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.1100000000000003},"max":{"1min":46.259999999999998,"5min":77.727999999999994,"15min":222.346},"last":14.859999999999999},{"interface":"front","average":{"1min":21.076000000000001,"5min":18.771999999999998,"15min":19.385000000000002},"min":{"1min":7.4039999999999999,"5min":7.4039999999999999,"15min":6.0519999999999996},"max":{"1min":53.518000000000001,"5min":137.86799999999999,"15min":216.02000000000001},"last":16.192}]},{"osd":2,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":17.227,"5min":16.443999999999999,"15min":18.135000000000002},"min":{"1min":7.0960000000000001,"5min":6.173,"15min":6.0970000000000004},"max":{"1min":53.883000000000003,"5min":171.38399999999999,"15min":259.274},"last":6.8659999999999997},{"interface":"front","average":{"1min":26.795000000000002,"5min":20.93,"15min":21.888000000000002},"min":{"1min":9.859,"5min":7.6989999999999998,"15min":7.6989999999999998},"max":{"1min":137.751,"5min":170.02600000000001,"15min":223.89699999999999},"last":7.665}]},{"osd":3,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":21.032,"5min":21.359999999999999,"15min":24.524999999999999},"min":{"1min":9.3279999999999994,"5min":7.6630000000000003,"15min":7.0609999999999999},"max":{"1min":51.694000000000003,"5min":208.94800000000001,"15min":352.79500000000002},"last":12.548999999999999},{"interface":"front","average":{"1min":20.038,"5min":22.895,"15min":22.268000000000001},"min":{"1min":10.686,"5min":9.7870000000000008,"15min":7.4370000000000003},"max":{"1min":46.271999999999998,"5min":178.21600000000001,"15min":193.483},"last":14.077}]},{"osd":4,"last update":"Tue Aug 22 23:45:01 2023","interfaces":[{"interface":"back","average":{"1min":16.306999999999999,"5min":16.852,"15min":19.088999999999999},"min":{"1min":8.1959999999999997,"5min":6.8609999999999998,"15min":6.2069999999999999},"max":{"1min":40.009999999999998,"5min":91.307000000000002,"15min":151.41399999999999},"last":11.217000000000001},{"interface":"front","average":{"1min":22.344000000000001,"5min":22.126000000000001,"15min":26.108000000000001},"min":{"1min":10.922000000000001,"5min":9.2650000000000006,"15min":9.2650000000000006},"max":{"1min":50.005000000000003,"5min":92.557000000000002,"15min":156.40100000000001},"last":15.496}]},{"osd":5,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":23.013999999999999,"5min":22.649999999999999,"15min":25.038},"min":{"1min":11.438000000000001,"5min":7.7779999999999996,"15min":7.2290000000000001},"max":{"1min":77.650000000000006,"5min":183.268,"15min":249.37299999999999},"last":12.244},{"interface":"front","average":{"1min":21.149999999999999,"5min":21.687000000000001,"15min":23.161999999999999},"min":{"1min":8.9149999999999991,"5min":8.9149999999999991,"15min":7.3849999999999998},"max":{"1min":62.554000000000002,"5min":146.96199999999999,"15min":215.76499999999999},"last":19.224}]},{"osd":7,"last update":"Tue Aug 22 23:45:21 2023","interfaces":[{"interface":"back","average":{"1min":26.050999999999998,"5min":22.292000000000002,"15min":25.626999999999999},"min":{"1min":10.052,"5min":10.052,"15min":9.3559999999999999},"max":{"1min":82.512,"5min":91.301000000000002,"15min":457.08199999999999},"last":20.658000000000001},{"interface":"front","average":{"1min":27.477,"5min":23.448,"15min":26.001999999999999},"min":{"1min":13.436999999999999,"5min":8.1170000000000009,"15min":8.1170000000000009},"max":{"1min":81.609999999999999,"5min":155.268,"15min":353.75900000000001},"last":16.835000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509723,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88296,"kb_used_data":12896,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94283544,"statfs":{"total":96636764160,"available":96546349056,"internally_reserved":0,"allocated":13205504,"data_stored":8072918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":9.5980000000000008,"5min":19.257999999999999,"15min":21.283000000000001},"min":{"1min":5.9900000000000002,"5min":4.1900000000000004,"15min":4.1900000000000004},"max":{"1min":17.904,"5min":141.732,"15min":235.06100000000001},"last":7.2750000000000004},{"interface":"front","average":{"1min":14.381,"5min":22.114999999999998,"15min":25.699000000000002},"min":{"1min":6.8630000000000004,"5min":5.04,"15min":4.5330000000000004},"max":{"1min":35.685000000000002,"5min":189.517,"15min":395.15300000000002},"last":13.686}]},{"osd":1,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":15.704000000000001,"5min":24.719000000000001,"15min":23.050000000000001},"min":{"1min":8.1270000000000007,"5min":6.7149999999999999,"15min":6.0039999999999996},"max":{"1min":46.531999999999996,"5min":241.87799999999999,"15min":347.69200000000001},"last":10.815},{"interface":"front","average":{"1min":20.838000000000001,"5min":32.209000000000003,"15min":29.765999999999998},"min":{"1min":11.089,"5min":10.084,"15min":7.1029999999999998},"max":{"1min":49.670000000000002,"5min":252.441,"15min":350.66199999999998},"last":14.385999999999999}]},{"osd":2,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":15.763,"5min":21.625,"15min":22.677},"min":{"1min":7.0289999999999999,"5min":6.2359999999999998,"15min":6.0970000000000004},"max":{"1min":47.415999999999997,"5min":219.37899999999999,"15min":234.05699999999999},"last":9.8130000000000006},{"interface":"front","average":{"1min":16.007999999999999,"5min":22.385000000000002,"15min":28.353000000000002},"min":{"1min":6.5629999999999997,"5min":6.5629999999999997,"15min":6.1210000000000004},"max":{"1min":37.369,"5min":214.50700000000001,"15min":467.80500000000001},"last":16.614999999999998}]},{"osd":3,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":16.234000000000002,"5min":22.823,"15min":26.353000000000002},"min":{"1min":8.1280000000000001,"5min":7.0449999999999999,"15min":6.899},"max":{"1min":55.274999999999999,"5min":161.05199999999999,"15min":284.47899999999998},"last":21.550000000000001},{"interface":"front","average":{"1min":18.247,"5min":28.542999999999999,"15min":29.581},"min":{"1min":8.3350000000000009,"5min":7.9429999999999996,"15min":6.9870000000000001},"max":{"1min":40.488999999999997,"5min":226.25700000000001,"15min":339.51499999999999},"last":15.105}]},{"osd":5,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":28.321000000000002,"5min":31.291,"15min":30.677},"min":{"1min":9.5660000000000007,"5min":8.5909999999999993,"15min":7.2279999999999998},"max":{"1min":155.17599999999999,"5min":238.57300000000001,"15min":380.863},"last":15.779999999999999},{"interface":"front","average":{"1min":27.881,"5min":29.675000000000001,"15min":33.774000000000001},"min":{"1min":10.247999999999999,"5min":7.2400000000000002,"15min":7.2400000000000002},"max":{"1min":158.03899999999999,"5min":215.928,"15min":342.858},"last":11.561999999999999}]},{"osd":6,"last update":"Tue Aug 22 23:45:13 2023","interfaces":[{"interface":"back","average":{"1min":23.414000000000001,"5min":29.004999999999999,"15min":30.018999999999998},"min":{"1min":12.186999999999999,"5min":10.523999999999999,"15min":7.1219999999999999},"max":{"1min":70.799000000000007,"5min":217,"15min":249.29599999999999},"last":17.323},{"interface":"front","average":{"1min":19.931000000000001,"5min":23.629000000000001,"15min":28.702999999999999},"min":{"1min":7.6280000000000001,"5min":7.5060000000000002,"15min":7.1870000000000003},"max":{"1min":74.850999999999999,"5min":220.87100000000001,"15min":468.887},"last":18.946999999999999}]},{"osd":7,"last update":"Tue Aug 22 23:45:17 2023","interfaces":[{"interface":"back","average":{"1min":18.567,"5min":22.748999999999999,"15min":28.390000000000001},"min":{"1min":9.4450000000000003,"5min":8.0540000000000003,"15min":8.0540000000000003},"max":{"1min":52.113999999999997,"5min":187.30099999999999,"15min":333.20699999999999},"last":19.777999999999999},{"interface":"front","average":{"1min":22.135000000000002,"5min":26.247,"15min":29.818000000000001},"min":{"1min":12.84,"5min":9.9369999999999994,"15min":7.5570000000000004},"max":{"1min":56.811,"5min":197.05600000000001,"15min":368.08699999999999},"last":20.454000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542430,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87640,"kb_used_data":12240,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94284200,"statfs":{"total":96636764160,"available":96547020800,"internally_reserved":0,"allocated":12533760,"data_stored":7553009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":77199627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":11.638,"5min":16.164999999999999,"15min":19.459},"min":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":4.3040000000000003},"max":{"1min":19.277999999999999,"5min":307.584,"15min":307.584},"last":7.0590000000000002},{"interface":"front","average":{"1min":15.122,"5min":17.684999999999999,"15min":20.268999999999998},"min":{"1min":8.9990000000000006,"5min":6.3029999999999999,"15min":5.0679999999999996},"max":{"1min":23.640999999999998,"5min":261.58999999999997,"15min":317.81},"last":16.594999999999999}]},{"osd":1,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":12.972,"5min":14.698,"15min":23.751999999999999},"min":{"1min":6.819,"5min":5.077,"15min":5.077},"max":{"1min":22.248000000000001,"5min":79.575000000000003,"15min":319.34500000000003},"last":9.2469999999999999},{"interface":"front","average":{"1min":11.390000000000001,"5min":12.893000000000001,"15min":19.420000000000002},"min":{"1min":6.96,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":17.920000000000002,"5min":78.543999999999997,"15min":195.78999999999999},"last":10.74}]},{"osd":2,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":16.155000000000001,"5min":16.762,"15min":20.832999999999998},"min":{"1min":8.6720000000000006,"5min":6.6879999999999997,"15min":5.7160000000000002},"max":{"1min":21.527000000000001,"5min":148.97800000000001,"15min":304.25},"last":18.943999999999999},{"interface":"front","average":{"1min":13.529999999999999,"5min":17.765000000000001,"15min":21.911000000000001},"min":{"1min":7.4160000000000004,"5min":7.3120000000000003,"15min":6.8339999999999996},"max":{"1min":21.960999999999999,"5min":151.46700000000001,"15min":264.49400000000003},"last":13.584}]},{"osd":3,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":11.039,"5min":17.289000000000001,"15min":21.934000000000001},"min":{"1min":7.7130000000000001,"5min":7.4740000000000002,"15min":6.5510000000000002},"max":{"1min":16.832999999999998,"5min":154.44200000000001,"15min":243.33699999999999},"last":11.497},{"interface":"front","average":{"1min":16.942,"5min":21.024999999999999,"15min":26.138999999999999},"min":{"1min":7.3049999999999997,"5min":6.7880000000000003,"15min":6.7750000000000004},"max":{"1min":30.774000000000001,"5min":246.798,"15min":337.09699999999998},"last":20.626000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":19.015000000000001,"15min":27.401},"min":{"1min":7.6020000000000003,"5min":7.4480000000000004,"15min":7.4480000000000004},"max":{"1min":23.989999999999998,"5min":111.015,"15min":265.17899999999997},"last":15.726000000000001},{"interface":"front","average":{"1min":15.244999999999999,"5min":17.884,"15min":22.994},"min":{"1min":11.186,"5min":7.2350000000000003,"15min":6.8419999999999996},"max":{"1min":21.315999999999999,"5min":117.06100000000001,"15min":192.58600000000001},"last":12.205}]},{"osd":5,"last update":"Tue Aug 22 23:45:06 2023","interfaces":[{"interface":"back","average":{"1min":15.492000000000001,"5min":20.783000000000001,"15min":28.716999999999999},"min":{"1min":11.125999999999999,"5min":8.4779999999999998,"15min":7.9889999999999999},"max":{"1min":23.506,"5min":247.898,"15min":342.53100000000001},"last":14.493},{"interface":"front","average":{"1min":18.222000000000001,"5min":21.045000000000002,"15min":26.207999999999998},"min":{"1min":10.242000000000001,"5min":8.8770000000000007,"15min":8.5809999999999995},"max":{"1min":33.665999999999997,"5min":135.31399999999999,"15min":271.47500000000002},"last":17.367000000000001}]},{"osd":6,"last update":"Tue Aug 22 23:45:18 2023","interfaces":[{"interface":"back","average":{"1min":18.920000000000002,"5min":19.762,"15min":24.591000000000001},"min":{"1min":12.364000000000001,"5min":7.4459999999999997,"15min":6.5679999999999996},"max":{"1min":24.834,"5min":144.21600000000001,"15min":228.43100000000001},"last":18.103999999999999},{"interface":"front","average":{"1min":19.818000000000001,"5min":20.763000000000002,"15min":27.73},"min":{"1min":8.8559999999999999,"5min":8.8559999999999999,"15min":8.8559999999999999},"max":{"1min":66.206000000000003,"5min":99.783000000000001,"15min":262.45800000000003},"last":21.582999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542427,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83936,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94287904,"statfs":{"total":96636764160,"available":96550813696,"internally_reserved":0,"allocated":12935168,"data_stored":7950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":73005323},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":19.690000000000001,"5min":17.018000000000001,"15min":19.709},"min":{"1min":6.0819999999999999,"5min":5.0819999999999999,"15min":3.9889999999999999},"max":{"1min":169.48699999999999,"5min":169.48699999999999,"15min":220.017},"last":13.746},{"interface":"front","average":{"1min":25.696999999999999,"5min":23.117000000000001,"15min":25.722000000000001},"min":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":5.9329999999999998},"max":{"1min":168.82400000000001,"5min":168.82400000000001,"15min":264.11900000000003},"last":13.077}]},{"osd":1,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":14.484,"5min":14.842000000000001,"15min":17.66},"min":{"1min":6.5830000000000002,"5min":6.5119999999999996,"15min":5.9379999999999997},"max":{"1min":40.689,"5min":102.66200000000001,"15min":257.60199999999998},"last":18.422999999999998},{"interface":"front","average":{"1min":22.949999999999999,"5min":21.742000000000001,"15min":23.655000000000001},"min":{"1min":7.0999999999999996,"5min":5.7400000000000002,"15min":5.7400000000000002},"max":{"1min":150.47300000000001,"5min":150.47300000000001,"15min":230.73400000000001},"last":12.371}]},{"osd":2,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":20.837,"5min":17.885999999999999,"15min":19.574000000000002},"min":{"1min":7.8120000000000003,"5min":6.4080000000000004,"15min":5.5609999999999999},"max":{"1min":82.043000000000006,"5min":116.45,"15min":248.74799999999999},"last":9.7550000000000008},{"interface":"front","average":{"1min":18.931000000000001,"5min":17.198,"15min":22.306999999999999},"min":{"1min":7.9569999999999999,"5min":6.8019999999999996,"15min":6.0919999999999996},"max":{"1min":82.939999999999998,"5min":99.498000000000005,"15min":256.94400000000002},"last":21.946999999999999}]},{"osd":3,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":22.113,"5min":25,"15min":29.079000000000001},"min":{"1min":9.8559999999999999,"5min":6.9859999999999998,"15min":6.5739999999999998},"max":{"1min":71.793999999999997,"5min":231.99799999999999,"15min":257.23500000000001},"last":11.634},{"interface":"front","average":{"1min":26.582999999999998,"5min":26.600000000000001,"15min":23.663},"min":{"1min":9.6709999999999994,"5min":6.827,"15min":6.7489999999999997},"max":{"1min":132.06299999999999,"5min":273.66000000000003,"15min":273.66000000000003},"last":14.473000000000001}]},{"osd":4,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":22.515000000000001,"5min":23.562000000000001,"15min":25.693000000000001},"min":{"1min":10.301,"5min":8.5860000000000003,"15min":6.8049999999999997},"max":{"1min":71.150000000000006,"5min":143.23500000000001,"15min":212.77500000000001},"last":16.146000000000001},{"interface":"front","average":{"1min":21.497,"5min":23.329999999999998,"15min":24.059999999999999},"min":{"1min":10.753,"5min":7.5389999999999997,"15min":7.2599999999999998},"max":{"1min":53.722999999999999,"5min":156.947,"15min":273.58699999999999},"last":8.4570000000000007}]},{"osd":6,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":29.331,"5min":28.152000000000001,"15min":30.099},"min":{"1min":9.0649999999999995,"5min":7.8019999999999996,"15min":7.8019999999999996},"max":{"1min":149.81399999999999,"5min":228.626,"15min":241.26499999999999},"last":15.298999999999999},{"interface":"front","average":{"1min":28.795000000000002,"5min":25.315000000000001,"15min":25.981999999999999},"min":{"1min":10.907,"5min":6.1660000000000004,"15min":6.1660000000000004},"max":{"1min":121.235,"5min":247.06800000000001,"15min":247.06800000000001},"last":16.859000000000002}]},{"osd":7,"last update":"Tue Aug 22 23:45:19 2023","interfaces":[{"interface":"back","average":{"1min":26.649000000000001,"5min":22.806000000000001,"15min":25.640000000000001},"min":{"1min":11.548999999999999,"5min":8.6859999999999999,"15min":8.2560000000000002},"max":{"1min":97.043999999999997,"5min":121.524,"15min":317.44600000000003},"last":17.588999999999999},{"interface":"front","average":{"1min":24.292000000000002,"5min":25.937999999999999,"15min":26.905999999999999},"min":{"1min":10.48,"5min":7.2960000000000003,"15min":7.2539999999999996},"max":{"1min":69.792000000000002,"5min":201.81700000000001,"15min":361.93000000000001},"last":19.166}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-22T23:45:53.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-08-22T23:45:53.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.484+0000 7ffa95cc5640 10 monclient: get_monmap_and_config 2023-08-22T23:45:53.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.484+0000 7ffa95cc5640 10 monclient: build_initial_monmap 2023-08-22T23:45:53.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.484+0000 7ffa95cc5640 10 monclient: monmap: 2023-08-22T23:45:53.486 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:53.486 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:53.486 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:45:53.485457+0000 2023-08-22T23:45:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:45:53.485457+0000 2023-08-22T23:45:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:53.488 INFO:teuthology.orchestra.run.smithi029.stderr:0: v1:172.21.15.29:6789/0 mon.noname-a-legacy 2023-08-22T23:45:53.488 INFO:teuthology.orchestra.run.smithi029.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2023-08-22T23:45:53.488 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-b 2023-08-22T23:45:53.488 INFO:teuthology.orchestra.run.smithi029.stderr:3: v2:172.21.15.29:3300/0 mon.noname-a 2023-08-22T23:45:53.489 INFO:teuthology.orchestra.run.smithi029.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2023-08-22T23:45:53.489 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:53.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.484+0000 7ffa95cc5640 1 Processor -- start 2023-08-22T23:45:53.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.484+0000 7ffa95cc5640 1 -- start start 2023-08-22T23:45:53.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.484+0000 7ffa95cc5640 10 monclient: init 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ffa901448e0 0x7ffa90144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa901448e0 addr v2:172.21.15.52:3300/0 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa900add90 0x7ffa90145200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa900add90 addr v2:172.21.15.29:3300/0 2023-08-22T23:45:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa9013e190 addr v1:172.21.15.29:6789/0 2023-08-22T23:45:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa900b12d0 con 0x7ffa9013e190 2023-08-22T23:45:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa900afae0 con 0x7ffa900add90 2023-08-22T23:45:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa900afdc0 con 0x7ffa901448e0 2023-08-22T23:45:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.485+0000 7ffa95cc5640 10 monclient(hunting): authenticate will time out at 2917.736010s 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa900add90 0x7ffa90145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8f7fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ffa901448e0 0x7ffa90144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa900add90 0x7ffa90145200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51270/0 (socket says 172.21.15.29:51270) 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 1 -- 172.21.15.29:0/1710368152 learned_addr learned my addr 172.21.15.29:0/1710368152 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 10 monclient(hunting): get_auth_request con 0x7ffa900add90 auth_method 0 2023-08-22T23:45:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa901448e0 auth_method 0 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.486+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.487+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.487+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.487+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.487+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 10 monclient(hunting): handle_auth_done global_id 14856 payload 274 2023-08-22T23:45:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 1 -- 172.21.15.29:0/1710368152 >> v2:172.21.15.52:3300/0 conn(0x7ffa901448e0 msgr2=0x7ffa90144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:45:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 1 --2- 172.21.15.29:0/1710368152 >> v2:172.21.15.52:3300/0 conn(0x7ffa901448e0 0x7ffa90144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 1 -- 172.21.15.29:0/1710368152 >> v1:172.21.15.29:6789/0 conn(0x7ffa9013e190 legacy=0x7ffa901410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-22T23:45:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 10 monclient: _finish_hunting 0 2023-08-22T23:45:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 1 monclient: found mon.noname-a 2023-08-22T23:45:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:45:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.488+0000 7ffa8effd640 1 -- 172.21.15.29:0/1710368152 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa900b00a0 con 0x7ffa900add90 2023-08-22T23:45:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.491+0000 7ffa8effd640 1 --2- 172.21.15.29:0/1710368152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa90145200 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7ffa900b7890 tx=0x7ffa8002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8078c5daf02b56a server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:53.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 1 -- 172.21.15.29:0/1710368152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa800317e0 con 0x7ffa900add90 2023-08-22T23:45:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:45:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: dump: 2023-08-22T23:45:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:53.507 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: _finish_auth 0 2023-08-22T23:45:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: _check_auth_tickets 2023-08-22T23:45:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 1 -- 172.21.15.29:0/1710368152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa80031940 con 0x7ffa900add90 2023-08-22T23:45:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 1 -- 172.21.15.29:0/1710368152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80031cd0 con 0x7ffa900add90 2023-08-22T23:45:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa95cc5640 5 monclient: authenticate success, global_id 14856 2023-08-22T23:45:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa95cc5640 10 monclient: get_monmap_and_config success 2023-08-22T23:45:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa95cc5640 4 set_mon_vals no callback set 2023-08-22T23:45:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:45:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.495+0000 7ffa8e7fc640 10 monclient: dump: 2023-08-22T23:45:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 10 monclient: shutdown 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 -- 172.21.15.29:0/1710368152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 msgr2=0x7ffa90145200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/1710368152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa90145200 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7ffa900b7890 tx=0x7ffa8002b880 comp rx=0 tx=0).stop 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 -- 172.21.15.29:0/1710368152 shutdown_connections 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/1710368152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa90145200 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/1710368152 >> v2:172.21.15.52:3300/0 conn(0x7ffa901448e0 0x7ffa90144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 -- 172.21.15.29:0/1710368152 >> 172.21.15.29:0/1710368152 conn(0x7ffa90065ed0 msgr2=0x7ffa900b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:53.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 -- 172.21.15.29:0/1710368152 shutdown_connections 2023-08-22T23:45:53.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 1 -- 172.21.15.29:0/1710368152 wait complete. 2023-08-22T23:45:53.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 10 monclient: build_initial_monmap 2023-08-22T23:45:53.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.496+0000 7ffa95cc5640 10 monclient: monmap: 2023-08-22T23:45:53.512 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 0 2023-08-22T23:45:53.512 INFO:teuthology.orchestra.run.smithi029.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-08-22T23:45:53.512 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 0.000000 2023-08-22T23:45:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:created 0.000000 2023-08-22T23:45:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 0 (unknown) 2023-08-22T23:45:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.noname-a 2023-08-22T23:45:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2023-08-22T23:45:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.noname-c 2023-08-22T23:45:53.514 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:53.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 1 librados: starting msgr at 2023-08-22T23:45:53.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 1 librados: starting objecter 2023-08-22T23:45:53.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 1 Processor -- start 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 1 -- start start 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 1 librados: setting wanted keys 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 1 librados: calling monclient init 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.497+0000 7ffa95cc5640 10 monclient: init 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient: _reopen_session rank -1 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient: _add_conns ranks=[0,2,1] 2023-08-22T23:45:53.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa9013eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa900add90 addr [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] 2023-08-22T23:45:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffa9013e190 0x7ffa90136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9013e190 addr [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2023-08-22T23:45:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa901448e0 0x7ffa90136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): picked mon.noname-b con 0x7ffa901448e0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2023-08-22T23:45:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa900a7830 con 0x7ffa900add90 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa900a74e0 con 0x7ffa901448e0 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): start opening mon connection 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 1 -- --> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901374e0 con 0x7ffa9013e190 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): _renew_subs 2023-08-22T23:45:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa95cc5640 10 monclient(hunting): authenticate will time out at 2917.748580s 2023-08-22T23:45:53.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa9013eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:53.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8effd640 1 --2- >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffa9013e190 0x7ffa90136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:53.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa9013eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51280/0 (socket says 172.21.15.29:51280) 2023-08-22T23:45:53.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 1 -- 172.21.15.29:0/3948681092 learned_addr learned my addr 172.21.15.29:0/3948681092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa900add90 auth_method 0 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.498+0000 7ffa8ffff640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa901448e0 0x7ffa90136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): get_auth_request con 0x7ffa9013e190 auth_method 0 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): get_auth_request con 0x7ffa901448e0 auth_method 0 2023-08-22T23:45:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-08-22T23:45:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): _init_auth method 2 2023-08-22T23:45:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): _init_auth creating new auth 2023-08-22T23:45:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-08-22T23:45:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-08-22T23:45:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.499+0000 7ffa8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-08-22T23:45:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_done global_id 14859 payload 824 2023-08-22T23:45:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 1 -- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffa9013e190 msgr2=0x7ffa90136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffa9013e190 0x7ffa90136a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 1 -- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa901448e0 msgr2=0x7ffa90136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa901448e0 0x7ffa90136fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:53.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 10 monclient: _finish_hunting 0 2023-08-22T23:45:53.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 1 monclient: found mon.noname-a 2023-08-22T23:45:53.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-08-22T23:45:53.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.29:3300/0 2023-08-22T23:45:53.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.502+0000 7ffa8f7fe640 1 -- 172.21.15.29:0/3948681092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800094d0 con 0x7ffa900add90 2023-08-22T23:45:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.505+0000 7ffa8f7fe640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa9013eef0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800e8a0 tx=0x7ffa7800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.505+0000 7ffa87fff640 10 client.?.objecter ms_handle_connect 0x7ffa900add90 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.505+0000 7ffa87fff640 10 client.?.objecter resend_mon_ops 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.508+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa780094f0 con 0x7ffa900add90 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.508+0000 7ffa87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.508+0000 7ffa87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.508+0000 7ffa87fff640 10 monclient: dump: 2023-08-22T23:45:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 10 monclient: _finish_auth 0 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 10 monclient: _check_auth_tickets 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7800c940 con 0x7ffa900add90 2023-08-22T23:45:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 5 monclient: authenticate success, global_id 14859 2023-08-22T23:45:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 monclient: _renew_subs 2023-08-22T23:45:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa901412d0 con 0x7ffa900add90 2023-08-22T23:45:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 10 monclient: handle_config config(0 keys) v1 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7800ccf0 con 0x7ffa900add90 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa87fff640 10 monclient: dump: 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:epoch 1 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:fsid cfd0f96d-1c62-4667-a308-870e3273ade2 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:last_changed 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:created 2023-08-22T23:20:45.339857+0000 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:min_mon_release 18 (reef) 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:election_strategy: 1 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2: [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon.c 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:45:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa8dffb640 4 set_mon_vals no callback set 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 client.14859.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 monclient: _renew_subs 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa90141e50 con 0x7ffa900add90 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 1 librados: init done 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:53.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:53.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.509+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa900a8b00 con 0x7ffa900add90 2023-08-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.529+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ffa78013470 con 0x7ffa900add90 2023-08-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.529+0000 7ffa87fff640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffa7006ba30 0x7ffa7006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.530+0000 7ffa8effd640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffa7006ba30 0x7ffa7006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.530+0000 7ffa8effd640 10 monclient: get_auth_request con 0x7ffa7006ba30 auth_method 0 2023-08-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.530+0000 7ffa8effd640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffa7006ba30 0x7ffa7006def0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffa80036000 tx=0x7ffa8003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.530+0000 7ffa87fff640 10 client.14859.objecter ms_handle_connect 0x7ffa7006ba30 2023-08-22T23:45:53.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(918..918 src has 827..918) v4 ==== 6137+0+0 (secure 0 0 0) 0x7ffa7805bf10 con 0x7ffa900add90 2023-08-22T23:45:53.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 10 client.14859.objecter ms_dispatch 0x7ffa90000b90 osd_map(918..918 src has 827..918) v4 2023-08-22T23:45:53.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 3 client.14859.objecter handle_osd_map got epochs [918,918] > 0 2023-08-22T23:45:53.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 3 client.14859.objecter handle_osd_map decoding full epoch 918 2023-08-22T23:45:53.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 20 client.14859.objecter dump_active .. 0 homeless 2023-08-22T23:45:53.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178480 (secure 0 0 0) 0x7ffa780b3940 con 0x7ffa900add90 2023-08-22T23:45:53.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-08-22T23:45:53.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.792+0000 7ffa87fff640 10 monclient: _finish_command 1 = system:0 2023-08-22T23:45:53.867 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:39.151 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.873 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:39.157 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.886 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:39.170 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.922 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:39.206 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.922 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:39.219 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.923 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:39.207 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.925 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:39.223 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.934 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:39.218 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.936 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:39.234 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:45:53.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.979+0000 7ffa95cc5640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-08-22T23:45:53.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.979+0000 7ffa95cc5640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-08-22T23:45:53.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.979+0000 7ffa95cc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.29:3300/0 2023-08-22T23:45:53.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:53.979+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ffa90066cb0 con 0x7ffa900add90 2023-08-22T23:45:54.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ffa78054540 con 0x7ffa900add90 2023-08-22T23:45:54.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa87fff640 10 monclient: handle_config config(1 keys) v1 2023-08-22T23:45:54.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa8dffb640 4 set_mon_vals no callback set 2023-08-22T23:45:54.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa8dffb640 10 set_mon_vals mon_health_to_clog = false 2023-08-22T23:45:54.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa87fff640 1 -- 172.21.15.29:0/3948681092 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7ffa780210c0 con 0x7ffa900add90 2023-08-22T23:45:54.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-08-22T23:45:54.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.316+0000 7ffa87fff640 10 monclient: _finish_command 2 = system:0 2023-08-22T23:45:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 10 librados: watch_flush enter 2023-08-22T23:45:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 10 librados: watch_flush exit 2023-08-22T23:45:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 20 client.14859.objecter shutdown clearing up homeless session... 2023-08-22T23:45:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 10 client.14859.objecter successfully canceled tick 2023-08-22T23:45:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffa7006ba30 msgr2=0x7ffa7006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffa7006ba30 0x7ffa7006def0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffa80036000 tx=0x7ffa8003d040 comp rx=0 tx=0).stop 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 10 monclient: shutdown 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 20 monclient: shutdown discarding 0 pending message(s) 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 msgr2=0x7ffa9013eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa9013eef0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800e8a0 tx=0x7ffa7800ed60 comp rx=0 tx=0).stop 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 shutdown_connections 2023-08-22T23:45:54.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:6800/697719510,v1:172.21.15.52:6801/697719510] conn(0x7ffa7006ba30 0x7ffa7006def0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa901448e0 0x7ffa90136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] conn(0x7ffa9013e190 0x7ffa90136a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.318+0000 7ffa95cc5640 1 --2- 172.21.15.29:0/3948681092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa900add90 0x7ffa9013eef0 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.319+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 >> 172.21.15.29:0/3948681092 conn(0x7ffa90065ed0 msgr2=0x7ffa90137f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.319+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 shutdown_connections 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.319+0000 7ffa95cc5640 1 -- 172.21.15.29:0/3948681092 wait complete. 2023-08-22T23:45:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-08-22T23:45:54.319+0000 7ffa95cc5640 1 librados: shutdown 2023-08-22T23:45:54.331 INFO:teuthology.misc:Shutting down mds daemons... 2023-08-22T23:45:54.332 INFO:teuthology.misc:Shutting down osd daemons... 2023-08-22T23:45:54.332 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-08-22T23:45:54.333 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:45:54.351 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:45:54.337+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 87018) UID: 0 2023-08-22T23:45:54.354 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-08-22T23:45:54.339+0000 b47d640 -1 osd.0 918 *** Got signal Terminated *** 2023-08-22T23:45:58.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-08-22T23:46:00.435 INFO:tasks.ceph.osd.0:Stopped 2023-08-22T23:46:00.435 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-08-22T23:46:00.436 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:46:00.451 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:46:00.440+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 87021) UID: 0 2023-08-22T23:46:00.451 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-08-22T23:46:00.442+0000 b47d640 -1 osd.1 920 *** Got signal Terminated *** 2023-08-22T23:46:03.888 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:49.172 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.892 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:49.177 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.919 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:49.202 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.942 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:49.226 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.944 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:49.242 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.945 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:49.229 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.947 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:49.244 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.965 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:49.263 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:03.967 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:49.250 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:04.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-08-22T23:46:12.640 INFO:tasks.ceph.osd.1:Stopped 2023-08-22T23:46:12.640 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-08-22T23:46:12.640 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:46:12.656 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:46:12.643+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 87020) UID: 0 2023-08-22T23:46:12.657 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-08-22T23:46:12.645+0000 b47d640 -1 osd.2 922 *** Got signal Terminated *** 2023-08-22T23:46:14.022 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:59.305 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.025 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:59.308 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.044 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:24:59.328 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.073 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:59.370 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.073 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:59.371 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.086 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:59.384 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.093 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:59.376 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.094 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:59.378 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:14.111 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:24:59.395 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:16.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-08-22T23:46:18.746 INFO:tasks.ceph.osd.2:Stopped 2023-08-22T23:46:18.747 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-08-22T23:46:18.747 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:46:18.760 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:46:18.750+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 87024) UID: 0 2023-08-22T23:46:18.762 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-08-22T23:46:18.751+0000 b47d640 -1 osd.3 925 *** Got signal Terminated *** 2023-08-22T23:46:22.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-08-22T23:46:22.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-08-22T23:46:23.982 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:09.266 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:23.984 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:09.268 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.000 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:09.284 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.028 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:09.312 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.032 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:09.316 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.039 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:09.336 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.041 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:09.339 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.051 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:09.334 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.056 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:09.354 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:24.849 INFO:tasks.ceph.osd.3:Stopped 2023-08-22T23:46:24.850 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-08-22T23:46:24.850 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:46:24.863 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:46:24.854+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86887) UID: 0 2023-08-22T23:46:24.864 INFO:tasks.ceph.osd.4.smithi052.stderr:2023-08-22T23:46:24.855+0000 b47d640 -1 osd.4 928 *** Got signal Terminated *** 2023-08-22T23:46:28.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-08-22T23:46:33.957 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:19.241 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.008 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:19.292 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.026 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:19.310 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.052 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:19.349 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.056 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:19.354 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.061 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:19.346 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.063 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:19.347 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.071 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:19.369 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:34.076 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:19.360 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:43.987 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:29.270 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.034 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:29.331 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.034 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:29.332 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.042 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:29.326 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.047 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:29.345 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.058 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:29.342 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.088 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:29.372 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.089 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:29.374 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:44.102 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:29.385 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.076 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:39.359 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.078 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:39.362 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.097 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:39.381 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.121 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:39.419 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.124 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:39.422 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.126 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:39.410 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.130 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:39.414 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.140 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:39.438 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:46:54.152 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:39.436 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.103 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:49.386 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.105 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:49.389 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.123 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:49.406 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.151 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:49.449 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.154 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:49.450 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.157 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:49.439 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.158 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:49.442 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.165 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:49.463 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:04.174 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:49.458 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.128 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:59.412 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.133 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:59.417 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.150 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:25:59.434 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.180 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:59.478 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.181 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:59.464 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.184 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:59.468 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.184 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:59.482 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.198 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:25:59.482 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:14.199 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:59.496 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:19.766 INFO:tasks.ceph.osd.4:Stopped 2023-08-22T23:47:19.767 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-08-22T23:47:19.767 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:19.781 INFO:tasks.ceph.osd.5.smithi052.stderr:2023-08-22T23:47:19.771+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86889) UID: 0 2023-08-22T23:47:19.781 INFO:tasks.ceph.osd.5.smithi052.stderr:2023-08-22T23:47:19.772+0000 b47d640 -1 osd.5 930 *** Got signal Terminated *** 2023-08-22T23:47:24.169 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:09.453 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.172 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:09.456 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.187 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:26:09.471 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.188 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:09.471 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.189 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:26:09.473 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.205 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:26:09.489 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.217 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:09.515 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.221 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:09.519 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:24.237 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:09.535 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:25.869 INFO:tasks.ceph.osd.5:Stopped 2023-08-22T23:47:25.869 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-08-22T23:47:25.870 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:25.883 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:47:25.873+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86891) UID: 0 2023-08-22T23:47:25.883 INFO:tasks.ceph.osd.6.smithi052.stderr:2023-08-22T23:47:25.874+0000 b47d640 -1 osd.6 932 *** Got signal Terminated *** 2023-08-22T23:47:31.972 INFO:tasks.ceph.osd.6:Stopped 2023-08-22T23:47:31.973 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-08-22T23:47:31.973 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:31.987 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:47:31.977+0000 b47d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86893) UID: 0 2023-08-22T23:47:31.987 INFO:tasks.ceph.osd.7.smithi052.stderr:2023-08-22T23:47:31.978+0000 b47d640 -1 osd.7 932 *** Got signal Terminated *** 2023-08-22T23:47:34.143 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:19.426 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.194 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:19.477 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.210 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:19.494 86734== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.241 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:26:19.525 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.243 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:26:19.527 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.251 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:19.549 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.254 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:19.552 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.261 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:26:19.545 86738== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:34.269 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:19.566 86277== Warning: unimplemented fcntl command: 1036 2023-08-22T23:47:38.075 INFO:tasks.ceph.osd.7:Stopped 2023-08-22T23:47:38.076 INFO:teuthology.misc:Shutting down mgr daemons... 2023-08-22T23:47:38.076 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-08-22T23:47:38.077 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:38.104 INFO:tasks.ceph.mgr.y:Stopped 2023-08-22T23:47:38.104 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-08-22T23:47:38.105 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:38.105 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:47:38.105+0000 7f4de9527640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 86275) UID: 0 2023-08-22T23:47:38.106 INFO:tasks.ceph.mgr.x.smithi052.stderr:2023-08-22T23:47:38.105+0000 7f4de9527640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-08-22T23:47:38.134 INFO:tasks.ceph.mgr.x:Stopped 2023-08-22T23:47:38.134 INFO:teuthology.misc:Shutting down mon daemons... 2023-08-22T23:47:38.134 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-08-22T23:47:38.135 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:38.159 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-08-22T23:47:38.149+0000 9f3d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86700) UID: 0 2023-08-22T23:47:38.160 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-08-22T23:47:38.150+0000 9f3d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-08-22T23:47:38.809 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:24.093 86734== 2023-08-22T23:47:38.810 INFO:tasks.ceph.mon.a.smithi029.stderr:==00:00:26:24.093 86734== Exit program on first error (--exit-on-first-error=yes) 2023-08-22T23:47:38.827 INFO:tasks.ceph.mon.a.smithi029.stderr:daemon-helper: command failed with exit status 42 2023-08-22T23:47:43.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-08-22T23:47:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-08-22T23:47:44.237 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-08-22T23:47:44.238 INFO:tasks.ceph.mon.a:Stopped 2023-08-22T23:47:44.238 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-08-22T23:47:44.239 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:47:44.254 INFO:tasks.ceph.mon.c.smithi029.stderr:2023-08-22T23:47:44.243+0000 9f3d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86702) UID: 0 2023-08-22T23:47:44.255 INFO:tasks.ceph.mon.c.smithi029.stderr:2023-08-22T23:47:44.244+0000 9f3d640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-08-22T23:47:48.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-08-22T23:49:37.920 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:28:23.204 86738== 2023-08-22T23:49:37.920 INFO:tasks.ceph.mon.c.smithi029.stderr:==00:00:28:23.204 86738== Exit program on first error (--exit-on-first-error=yes) 2023-08-22T23:49:37.936 INFO:tasks.ceph.mon.c.smithi029.stderr:daemon-helper: command failed with exit status 42 2023-08-22T23:49:38.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-08-22T23:49:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-08-22T23:49:40.178 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-08-22T23:49:40.179 INFO:tasks.ceph.mon.c:Stopped 2023-08-22T23:49:40.179 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-08-22T23:49:40.180 INFO:teuthology.orchestra.run:waiting for 300 2023-08-22T23:49:40.197 INFO:tasks.ceph.mon.b.smithi052.stderr:2023-08-22T23:49:40.183+0000 9f3d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86273) UID: 0 2023-08-22T23:49:40.197 INFO:tasks.ceph.mon.b.smithi052.stderr:2023-08-22T23:49:40.185+0000 9f3d640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-08-22T23:49:43.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-08-22T23:50:05.896 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:51.193 86277== 2023-08-22T23:50:05.896 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:51.193 86277== Exit program on first error (--exit-on-first-error=yes) 2023-08-22T23:50:05.912 INFO:tasks.ceph.mon.b.smithi052.stderr:daemon-helper: command failed with exit status 42 2023-08-22T23:50:08.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-08-22T23:50:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-08-22T23:50:10.690 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-08-22T23:50:10.692 INFO:tasks.ceph.mon.b:Stopped 2023-08-22T23:50:10.692 INFO:tasks.ceph:Checking cluster log for badness... 2023-08-22T23:50:10.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-08-22T23:50:10.753 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:50:10.754 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-08-22T23:50:10.927 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:50:10.928 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-22T23:50:11.025 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:50:11.026 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-08-22T23:50:11.158 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:50:11.159 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-08-22T23:50:11.259 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:50:11.259 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-08-22T23:50:11.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:50:11.414 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-08-22T23:50:11.530 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:50:11.530 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-08-22T23:50:11.601 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:50:11.602 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-08-22T23:50:11.666 INFO:tasks.ceph:Archiving mon data... 2023-08-22T23:50:11.669 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/data/mon.a.tgz 2023-08-22T23:50:11.669 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-08-22T23:50:11.687 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.GxptyEcSey 2023-08-22T23:50:11.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GxptyEcSey 2023-08-22T23:50:11.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.GxptyEcSey 2023-08-22T23:50:11.987 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.GxptyEcSey is 569KB 2023-08-22T23:50:12.093 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.GxptyEcSey 2023-08-22T23:50:12.109 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/data/mon.c.tgz 2023-08-22T23:50:12.110 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-08-22T23:50:12.162 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.L8e9tNu5AG 2023-08-22T23:50:12.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.L8e9tNu5AG 2023-08-22T23:50:12.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.L8e9tNu5AG 2023-08-22T23:50:12.435 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.L8e9tNu5AG is 569KB 2023-08-22T23:50:12.504 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.L8e9tNu5AG 2023-08-22T23:50:12.520 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/data/mon.b.tgz 2023-08-22T23:50:12.520 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2023-08-22T23:50:12.539 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.daNWKXFSbT 2023-08-22T23:50:12.539 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.daNWKXFSbT 2023-08-22T23:50:12.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.daNWKXFSbT 2023-08-22T23:50:12.837 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.daNWKXFSbT is 570KB 2023-08-22T23:50:12.934 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.daNWKXFSbT 2023-08-22T23:50:12.949 INFO:tasks.ceph:Cleaning ceph cluster... 2023-08-22T23:50:12.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-08-22T23:50:12.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-08-22T23:50:13.019 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-08-22T23:50:13.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-08-22T23:50:13.023 DEBUG:teuthology.orchestra.run.smithi052:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-08-22T23:50:13.070 INFO:teuthology.orchestra.run.smithi029.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-08-22T23:50:13.070 INFO:teuthology.orchestra.run.smithi029.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-08-22T23:50:13.071 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-08-22T23:50:13.071 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-08-22T23:50:13.074 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-08-22T23:50:13.074 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-08-22T23:50:13.105 INFO:teuthology.orchestra.run.smithi052.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-08-22T23:50:13.105 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-08-22T23:50:13.106 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-08-22T23:50:13.106 INFO:tasks.ceph:Archiving crash dumps... 2023-08-22T23:50:13.109 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/remote/smithi029/crash 2023-08-22T23:50:13.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-08-22T23:50:13.149 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/remote/smithi052/crash 2023-08-22T23:50:13.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-08-22T23:50:13.180 INFO:tasks.ceph:Compressing logs... 2023-08-22T23:50:13.181 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-22T23:50:13.194 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-22T23:50:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91669.log 2023-08-22T23:50:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90735.log 2023-08-22T23:50:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-08-22T23:50:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91669.log: /var/log/ceph/ceph-client.admin.90735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141191.log 2023-08-22T23:50:13.226 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91669.log.gz 2023-08-22T23:50:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92309.log 2023-08-22T23:50:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92451.log 2023-08-22T23:50:13.226 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90735.log.gz 2023-08-22T23:50:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92651.log 2023-08-22T23:50:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92309.log: /var/log/ceph/ceph-client.admin.92451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92591.log 2023-08-22T23:50:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92481.log 2023-08-22T23:50:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92915.log 2023-08-22T23:50:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90627.log 2023-08-22T23:50:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92915.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.141191.log.gz 2023-08-22T23:50:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93024.log 2023-08-22T23:50:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90627.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90627.log.gz 2023-08-22T23:50:13.229 INFO:teuthology.orchestra.run.smithi029.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92915.log.gz 2023-08-22T23:50:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-08-22T23:50:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158621.log 2023-08-22T23:50:13.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92651.log.gz 2023-08-22T23:50:13.232 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92933.log 2023-08-22T23:50:13.233 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158621.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92309.log.gz 2023-08-22T23:50:13.233 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158621.log.gz 2023-08-22T23:50:13.233 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159335.log 2023-08-22T23:50:13.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92933.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-08-22T23:50:13.236 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159335.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.92933.log.gz 2023-08-22T23:50:13.236 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93064.log 2023-08-22T23:50:13.236 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.159335.log.gz 2023-08-22T23:50:13.237 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92951.log 2023-08-22T23:50:13.237 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92591.log.gz 2023-08-22T23:50:13.237 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91377.log 2023-08-22T23:50:13.237 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93064.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.93024.log.gz/var/log/ceph/ceph-client.admin.92951.log: 2023-08-22T23:50:13.237 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118908.log 2023-08-22T23:50:13.238 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91377.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91377.log.gz 2023-08-22T23:50:13.238 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92698.log 2023-08-22T23:50:13.239 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118908.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92951.log.gz 2023-08-22T23:50:13.239 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91047.log 2023-08-22T23:50:13.242 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92698.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.93064.log.gz 2023-08-22T23:50:13.242 INFO:teuthology.orchestra.run.smithi029.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.118908.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92950.log 2023-08-22T23:50:13.242 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:50:13.242 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90825.log 2023-08-22T23:50:13.242 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91047.log: /var/log/ceph/ceph-client.admin.92950.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91047.log.gz 2023-08-22T23:50:13.243 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130815.log 2023-08-22T23:50:13.243 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90825.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.90825.log.gz 2023-08-22T23:50:13.243 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158747.log 2023-08-22T23:50:13.246 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130815.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92950.log.gz 2023-08-22T23:50:13.247 INFO:teuthology.orchestra.run.smithi029.stderr: 91.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159503.log 2023-08-22T23:50:13.247 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92698.log.gz 2023-08-22T23:50:13.247 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108401.log 2023-08-22T23:50:13.247 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159503.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.130815.log.gz 2023-08-22T23:50:13.248 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158747.log.gz 2023-08-22T23:50:13.248 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92257.log 2023-08-22T23:50:13.249 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158957.log 2023-08-22T23:50:13.249 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159503.log.gz 2023-08-22T23:50:13.249 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108401.log: /var/log/ceph/ceph-client.admin.92257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118913.log 2023-08-22T23:50:13.253 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158957.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158957.log.gz 2023-08-22T23:50:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91046.log 2023-08-22T23:50:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118913.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.108401.log.gz 2023-08-22T23:50:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91381.log 2023-08-22T23:50:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91046.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91046.log.gz 2023-08-22T23:50:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158999.log 2023-08-22T23:50:13.254 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91381.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91381.log.gz 2023-08-22T23:50:13.255 INFO:teuthology.orchestra.run.smithi029.stderr: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159083.log 2023-08-22T23:50:13.255 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118913.log.gz 2023-08-22T23:50:13.255 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104318.log 2023-08-22T23:50:13.260 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159083.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158999.log.gz 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.159083.log.gz 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87047.log 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159125.log 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104318.log: /var/log/ceph/ceph-client.admin.87047.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87047.log.gz 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90781.log 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159125.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159125.log.gz 2023-08-22T23:50:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92600.log 2023-08-22T23:50:13.262 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90781.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.104318.log.gz 2023-08-22T23:50:13.262 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90781.log.gz 2023-08-22T23:50:13.262 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91045.log 2023-08-22T23:50:13.262 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159377.log 2023-08-22T23:50:13.262 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92600.log: /var/log/ceph/ceph-client.admin.91045.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91045.log.gz 2023-08-22T23:50:13.262 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91043.log 2023-08-22T23:50:13.265 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159377.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159377.log.gz 2023-08-22T23:50:13.265 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104317.log 2023-08-22T23:50:13.265 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91043.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91043.log.gz 2023-08-22T23:50:13.266 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90588.log 2023-08-22T23:50:13.267 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86561.log 2023-08-22T23:50:13.268 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-08-22T23:50:13.268 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86561.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86561.log.gz 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86641.log 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86641.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-08-22T23:50:13.269 INFO:teuthology.orchestra.run.smithi052.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86641.log.gz 2023-08-22T23:50:13.270 INFO:teuthology.orchestra.run.smithi052.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz/var/log/ceph/valgrind/osd.7.log: 2023-08-22T23:50:13.270 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-08-22T23:50:13.270 INFO:teuthology.orchestra.run.smithi052.stderr: 63.8% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-08-22T23:50:13.270 INFO:teuthology.orchestra.run.smithi052.stderr:gzip/var/log/ceph/valgrind/osd.5.log: -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-08-22T23:50:13.270 INFO:teuthology.orchestra.run.smithi052.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-08-22T23:50:13.271 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-08-22T23:50:13.271 INFO:teuthology.orchestra.run.smithi052.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-08-22T23:50:13.271 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86472.log 2023-08-22T23:50:13.271 INFO:teuthology.orchestra.run.smithi052.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-08-22T23:50:13.271 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-08-22T23:50:13.271 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86681.log 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86472.log.gz/var/log/ceph/ceph-client.admin.86681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86601.log 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86521.log 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr: 87.8%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.admin.86681.log.gz 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86601.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-08-22T23:50:13.272 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86521.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86601.log.gz 2023-08-22T23:50:13.273 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-08-22T23:50:13.273 INFO:teuthology.orchestra.run.smithi052.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86521.log.gz 2023-08-22T23:50:13.273 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86721.log 2023-08-22T23:50:13.273 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-08-22T23:50:13.273 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86430.log 2023-08-22T23:50:13.274 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86721.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86721.log.gz 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-08-22T23:50:13.274 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:50:13.274 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.86430.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86430.log.gz 2023-08-22T23:50:13.280 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104317.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.104317.log.gz 2023-08-22T23:50:13.280 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159546.log 2023-08-22T23:50:13.280 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90588.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90588.log.gz 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92600.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158873.log 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130817.log 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158873.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159546.log.gz 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91883.log 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130817.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158873.log.gz 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92943.log 2023-08-22T23:50:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91883.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91883.log.gz 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158831.log 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92943.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.130817.log.gz 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108400.log 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158831.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.158831.log.gz 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91379.log 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108400.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108400.log.gz 2023-08-22T23:50:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91379.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91379.log.gz 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.3.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/mon.a.log: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-08-22T23:50:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.0.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-08-22T23:50:13.286 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-08-22T23:50:13.286 INFO:teuthology.orchestra.run.smithi029.stderr: 91.3%/var/log/ceph/valgrind/mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.92943.log.gz 2023-08-22T23:50:13.286 INFO:teuthology.orchestra.run.smithi029.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-08-22T23:50:13.286 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-08-22T23:50:13.286 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-08-22T23:50:13.287 INFO:teuthology.orchestra.run.smithi029.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-08-22T23:50:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90178.log 2023-08-22T23:50:13.287 INFO:teuthology.orchestra.run.smithi029.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz/var/log/ceph/ceph.audit.log: 2023-08-22T23:50:13.287 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93051.log 2023-08-22T23:50:13.288 INFO:teuthology.orchestra.run.smithi029.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.92257.log.gz 2023-08-22T23:50:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118909.log 2023-08-22T23:50:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93051.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90178.log.gz 2023-08-22T23:50:13.288 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158789.log 2023-08-22T23:50:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118909.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.118909.log.gz 2023-08-22T23:50:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92769.log 2023-08-22T23:50:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158789.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93051.log.gz 2023-08-22T23:50:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92326.log 2023-08-22T23:50:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92769.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158789.log.gz 2023-08-22T23:50:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159251.log 2023-08-22T23:50:13.301 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92326.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.92769.log.gz 2023-08-22T23:50:13.301 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91589.log 2023-08-22T23:50:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159251.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-08-22T23:50:13.302 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159251.log.gz 2023-08-22T23:50:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91925.log 2023-08-22T23:50:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141192.log 2023-08-22T23:50:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91925.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91589.log.gz 2023-08-22T23:50:13.304 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91925.log.gz 2023-08-22T23:50:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93053.log 2023-08-22T23:50:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108543.log 2023-08-22T23:50:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93053.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93053.log.gz 2023-08-22T23:50:13.312 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91042.log 2023-08-22T23:50:13.312 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108543.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108543.log.gz 2023-08-22T23:50:13.313 INFO:teuthology.orchestra.run.smithi029.stderr:gzip 90.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.91383.log 2023-08-22T23:50:13.313 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.141192.log.gz 2023-08-22T23:50:13.313 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91619.log 2023-08-22T23:50:13.313 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91383.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91042.log.gz 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91383.log.gz 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159209.log 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91828.log 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91619.log: /var/log/ceph/ceph-client.admin.159209.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91619.log.gz 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159209.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104316.log 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159588.log 2023-08-22T23:50:13.314 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91828.log: /var/log/ceph/ceph-client.admin.104316.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91828.log.gz 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91376.log 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159588.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159588.log.gz 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91048.log 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91376.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91376.log.gz 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91044.log 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.104316.log.gz 2023-08-22T23:50:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141190.log 2023-08-22T23:50:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91044.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91048.log.gz 2023-08-22T23:50:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91709.log 2023-08-22T23:50:13.317 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91044.log.gz 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141190.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.141190.log.gz 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92347.log 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92667.log 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92347.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91709.log.gz 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108399.log 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92667.log: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.92451.log.gz 2023-08-22T23:50:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92434.log 2023-08-22T23:50:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108399.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.92347.log.gz 2023-08-22T23:50:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159293.log 2023-08-22T23:50:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92434.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92667.log.gz 2023-08-22T23:50:13.321 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.108399.log.gz 2023-08-22T23:50:13.323 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92434.log.gz 2023-08-22T23:50:13.324 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92924.log 2023-08-22T23:50:13.324 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159628.log 2023-08-22T23:50:13.325 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92291.log 2023-08-22T23:50:13.325 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.159293.log.gz 2023-08-22T23:50:13.325 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158663.log 2023-08-22T23:50:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92291.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.159628.log.gz 2023-08-22T23:50:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80891.log 2023-08-22T23:50:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158663.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92924.log.gz 2023-08-22T23:50:13.328 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158663.log.gz 2023-08-22T23:50:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91766.log 2023-08-22T23:50:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.80891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158705.log 2023-08-22T23:50:13.329 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80891.log.gz 2023-08-22T23:50:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91766.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91766.log.gz 2023-08-22T23:50:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-08-22T23:50:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159167.log 2023-08-22T23:50:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158705.log: /var/log/ceph/ceph-mgr.y.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158705.log.gz 2023-08-22T23:50:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91967.log 2023-08-22T23:50:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159167.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159167.log.gz 2023-08-22T23:50:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92365.log 2023-08-22T23:50:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91967.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91967.log.gz 2023-08-22T23:50:13.334 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92291.log.gz 2023-08-22T23:50:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88877.log 2023-08-22T23:50:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.92365.log: /var/log/ceph/ceph-osd.3.log 2023-08-22T23:50:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.88877.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88877.log.gz 2023-08-22T23:50:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-08-22T23:50:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.3.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92365.log.gz 2023-08-22T23:50:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-08-22T23:50:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-08-22T23:50:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92921.log 2023-08-22T23:50:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.92326.log.gz 2023-08-22T23:50:13.363 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92368.log 2023-08-22T23:50:13.364 INFO:teuthology.orchestra.run.smithi029.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-08-22T23:50:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130816.log 2023-08-22T23:50:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130814.log 2023-08-22T23:50:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130816.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.130816.log.gz 2023-08-22T23:50:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91386.log 2023-08-22T23:50:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130814.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87902.log 2023-08-22T23:50:13.408 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.130814.log.gz 2023-08-22T23:50:13.408 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91386.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91386.log.gz 2023-08-22T23:50:13.415 INFO:teuthology.orchestra.run.smithi029.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92921.log.gz 2023-08-22T23:50:13.416 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158915.log 2023-08-22T23:50:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87902.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87902.log.gz 2023-08-22T23:50:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-08-22T23:50:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158915.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158915.log.gz 2023-08-22T23:50:13.419 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91769.log 2023-08-22T23:50:13.445 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92278.log 2023-08-22T23:50:13.445 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91769.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91769.log.gz 2023-08-22T23:50:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87072.log 2023-08-22T23:50:13.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159461.log 2023-08-22T23:50:13.460 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87072.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92809.log 2023-08-22T23:50:13.462 INFO:teuthology.orchestra.run.smithi052.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-08-22T23:50:13.466 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159461.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87072.log.gz 2023-08-22T23:50:13.474 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92760.log 2023-08-22T23:50:13.474 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92809.log: -- replaced with /var/log/ceph/ceph-client.admin.159461.log.gz 2023-08-22T23:50:13.481 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86742.log 2023-08-22T23:50:13.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92760.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91049.log 2023-08-22T23:50:13.489 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92809.log.gz 2023-08-22T23:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.86742.log: 91.4% 92.3% -- replaced with /var/log/ceph/ceph-client.admin.92760.log.gz 2023-08-22T23:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91389.log 2023-08-22T23:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86742.log.gz 2023-08-22T23:50:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91049.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91049.log.gz 2023-08-22T23:50:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93065.log 2023-08-22T23:50:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91389.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91389.log.gz 2023-08-22T23:50:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90683.log 2023-08-22T23:50:13.548 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93065.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.93065.log.gz 2023-08-22T23:50:13.550 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91385.log 2023-08-22T23:50:13.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90683.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90683.log.gz 2023-08-22T23:50:13.550 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92278.log.gz 2023-08-22T23:50:13.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-08-22T23:50:13.559 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92670.log 2023-08-22T23:50:13.560 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91385.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91385.log.gz 2023-08-22T23:50:13.574 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108539.log 2023-08-22T23:50:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92670.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92670.log.gz 2023-08-22T23:50:13.591 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87170.log 2023-08-22T23:50:13.594 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108539.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.108539.log.gz 2023-08-22T23:50:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130813.log 2023-08-22T23:50:13.607 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87170.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87170.log.gz 2023-08-22T23:50:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159041.log 2023-08-22T23:50:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90867.log 2023-08-22T23:50:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159041.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.130813.log.gz 2023-08-22T23:50:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159419.log 2023-08-22T23:50:13.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.90867.log: 88.8% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159041.log.gz 2023-08-22T23:50:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159419.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90867.log.gz 2023-08-22T23:50:13.652 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159419.log.gz 2023-08-22T23:50:13.761 INFO:teuthology.orchestra.run.smithi029.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.92481.log.gz 2023-08-22T23:50:14.431 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92368.log.gz 2023-08-22T23:50:14.873 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-08-22T23:50:15.023 INFO:teuthology.orchestra.run.smithi052.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-08-22T23:50:16.352 INFO:teuthology.orchestra.run.smithi029.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-08-22T23:50:17.063 INFO:teuthology.orchestra.run.smithi052.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-08-22T23:50:17.302 INFO:teuthology.orchestra.run.smithi052.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-08-22T23:50:17.308 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-08-22T23:50:17.324 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-08-22T23:50:17.573 INFO:teuthology.orchestra.run.smithi052.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-08-22T23:50:17.576 INFO:teuthology.orchestra.run.smithi052.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-08-22T23:50:17.576 INFO:teuthology.orchestra.run.smithi052.stderr: 2023-08-22T23:50:17.576 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m4.341s 2023-08-22T23:50:17.576 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m17.915s 2023-08-22T23:50:17.576 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.580s 2023-08-22T23:50:17.982 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-08-22T23:50:18.004 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-08-22T23:50:18.006 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-08-22T23:50:18.006 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m4.800s 2023-08-22T23:50:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m23.365s 2023-08-22T23:50:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.793s 2023-08-22T23:50:18.007 INFO:tasks.ceph:Archiving logs... 2023-08-22T23:50:18.007 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/remote/smithi029/log 2023-08-22T23:50:18.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-22T23:50:21.202 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/remote/smithi052/log 2023-08-22T23:50:21.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-22T23:50:23.641 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-08-22T23:50:23.642 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-08-22T23:50:23.651 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f0ed704638a5702969ce59fb4418946a60474648/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-08-22T23:50:23.652 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-08-22T23:50:23.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-22T23:50:23.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-22T23:50:23.755 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-08-22T23:50:23.765 INFO:teuthology.task.clock:Checking final clock skew... 2023-08-22T23:50:23.766 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-22T23:50:23.771 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-22T23:50:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-08-22T23:50:23.788 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-22T23:50:23.788 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-08-22T23:50:23.788 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +3077us[+2903us] +/- 99ms 2023-08-22T23:50:23.789 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1668us[-1842us] +/- 49ms 2023-08-22T23:50:23.789 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +88us[ -86us] +/- 65ms 2023-08-22T23:50:23.789 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:50:23.790 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2023-08-22T23:50:23.793 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-22T23:50:23.794 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2023-08-22T23:50:23.794 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +2991us[+2991us] +/- 99ms 2023-08-22T23:50:23.794 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1758us[-1829us] +/- 49ms 2023-08-22T23:50:23.794 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 34 +46us[ -26us] +/- 65ms 2023-08-22T23:50:23.794 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-22T23:50:23.795 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-08-22T23:50:23.809 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-08-22T23:50:23.809 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-08-22T23:50:23.819 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-22T23:50:23.941 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-22T23:50:24.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-22T23:50:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:50:24.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-22T23:50:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:50:24.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-08-22T23:50:24.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-08-22T23:50:24.274 INFO:teuthology.task.internal:Duration was 2857.848727 seconds 2023-08-22T23:50:24.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-08-22T23:50:24.284 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-08-22T23:50:24.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-22T23:50:24.287 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-22T23:50:24.341 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-22T23:50:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-22T23:50:24.667 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-08-22T23:50:24.668 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-08-22T23:50:24.669 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-22T23:50:24.699 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2023-08-22T23:50:24.700 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-22T23:50:24.735 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-08-22T23:50:24.736 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-22T23:50:24.744 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-22T23:50:24.814 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-08-22T23:50:24.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-22T23:50:24.827 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-22T23:50:25.443 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-08-22T23:50:25.452 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-22T23:50:25.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-22T23:50:25.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-22T23:50:25.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-08-22T23:50:25.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-22T23:50:25.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-22T23:50:25.560 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-08-22T23:50:25.587 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2023-08-22T23:50:25.624 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-22T23:50:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:50:25.640 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-22T23:50:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-22T23:50:25.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-08-22T23:50:25.701 INFO:teuthology.task.internal:Transferring archived files... 2023-08-22T23:50:25.703 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/remote/smithi029 2023-08-22T23:50:25.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-22T23:50:25.753 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793/remote/smithi052 2023-08-22T23:50:25.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-22T23:50:25.800 INFO:teuthology.task.internal:Removing archive directory... 2023-08-22T23:50:25.800 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-22T23:50:25.804 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-22T23:50:25.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-08-22T23:50:25.858 INFO:teuthology.task.internal:Not uploading archives. 2023-08-22T23:50:25.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-08-22T23:50:25.871 INFO:teuthology.task.internal:Tidying up after the test... 2023-08-22T23:50:25.871 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-22T23:50:25.874 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-22T23:50:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: 263923 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 22 23:50 /home/ubuntu/cephtest 2023-08-22T23:50:25.903 INFO:teuthology.orchestra.run.smithi052.stdout: 263230 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 22 23:50 /home/ubuntu/cephtest 2023-08-22T23:50:25.903 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-08-22T23:50:25.914 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-22T23:50:25.926 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-08-22T23:50:25.927 INFO:teuthology.nuke:Checking targets against current locks 2023-08-22T23:50:25.955 DEBUG:teuthology.nuke:shortname: smithi029 2023-08-22T23:50:25.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-22T23:50:25.985 DEBUG:teuthology.nuke:shortname: smithi052 2023-08-22T23:50:25.985 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-22T23:50:25.988 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-22 22:54:09.061993', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+3T+PSpbH3t890q4p+Mct7FKdX102eb77eXtY6aQVYryOmE5LO4b2qE6gLs6SdX09JTZKaLeLIpqbWMKtb1Vc='} 2023-08-22T23:50:26.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376793', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-22 22:54:09.060576', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG7NfRtW1+ZXWRHpP4aldS+1SQUMyv292c+czbs0gFMRfifl6nyklgZeVwlja+SpA4PHEyREN0RAP93I5k6s4eo='} 2023-08-22T23:50:26.056 INFO:teuthology.orchestra.console.smithi029:Power off 2023-08-22T23:50:26.058 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-22T23:50:26.085 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-08-22T23:50:26.086 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-22T23:50:26.104 INFO:teuthology.orchestra.console.smithi052:Power off 2023-08-22T23:50:26.105 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-22T23:50:26.115 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-08-22T23:50:26.128 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2023-08-22T23:50:26.128 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-22T23:50:26.155 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2023-08-22T23:50:30.117 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-22T23:50:30.157 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-22T23:50:30.234 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-08-22T23:50:30.271 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2023-08-22T23:50:34.236 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-22T23:50:34.273 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-22T23:50:34.350 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-08-22T23:50:34.385 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2023-08-22T23:50:34.452 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-08-22T23:50:34.487 INFO:teuthology.orchestra.console.smithi052:Power off completed 2023-08-22T23:50:34.593 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 2857.848727464676 failure_reason: saw valgrind issues flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-08-22T23:50:34.594 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-22T23:50:34.695 INFO:teuthology.run:FAIL